User interaction across cross-environment applications through an extended graphics context

ABSTRACT

Cross-environment rendering and user interaction support provide a seamless computing experience in a multi-operating system computing environment. The multi-operating system computing environment may include a mobile operating system and a desktop operating system running concurrently and independently on a shared kernel of a mobile computing device. User interaction support includes handling input events initially received in the shared kernel by accepting the input events in the desktop operating system and translating, mapping, and/or passing the input events through a virtual input device to the mobile operating system such that applications of the mobile operating system receive the input events as if coming from a user interaction space of the mobile operating system. The mobile computing device may be a smartphone running the Android mobile operating system and a full desktop Linux distribution on a modified Android kernel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a non-provisional of and claims the benefit of thefiling date of U.S. Provisional Application Nos. 61/389,117, filed Oct.1, 2010, entitled “Multi-Operating System Portable Docking Device”;61/507,199, filed Jul. 13, 2011, entitled “Dockable Mobile SoftwareArchitecture”; 61/507,201, filed Jul. 13, 2011, entitled“Cross-Environment Communication Framework”; 61/507,203, filed Jul. 13,2011, entitled “Multi-Operating System”; 61/507,206, filed Jul. 13,2011, entitled “Auto-Configuration of a Docked System in a Multi-OSEnvironment”; and 61/507,209, filed Jul. 13, 2011, entitled “Auto-Wakingof a Suspended Secondary OS in a Dockable System,” wherein the entirecontents of the foregoing priority applications are incorporated hereinby reference for all purposes.

BACKGROUND

1. Field

This application relates generally to the field of mobile computingenvironments, and more particularly to supporting multiple userenvironments through the use of multiple operating systems in a singlemobile computing device.

2. Relevant Background

Mobile computing devices are becoming ubiquitous in today's society. Forexample, as of the end of 2008, 90 percent of Americans had a mobilewireless device. At the same time, the capabilities of mobile devicesare advancing rapidly, including smartphones that integrate advancedcomputing capabilities with mobile telephony capabilities. Mobileproviders have launched hundreds of new smartphones in the last threeyears based upon several different platforms (e.g., Apple iPhone,Android, BlackBerry, Palm, and Windows Mobile). In the U.S., smartphonepenetration reached almost 23% by the middle of 2010, and over 35% insome age-groups. In Europe, the smartphone market grew by 41% from 2009to 2010, with over 60 million smartphone subscribers as of July 2010 inthe five largest European countries alone.

While smartphones are gaining in popularity and computing capability,they provide a limited user experience. Specifically, they typicallyhave an operating system that is modified for mobile device hardware anda restricted set of applications that are available for the modifiedoperating system. For example, many smartphones run Google's Androidoperating system. Android runs only applications that are specificallydeveloped to run within a Java-based virtual machine runtimeenvironment. In addition, while Android is based on a modified Linuxkernel, it uses different standard C libraries, system managers, andservices than Linux. Accordingly, applications written for Linux do notrun on Android without modification or porting. Similarly, Apple'siPhone uses the iOS mobile operating system. Again, while iOS is derivedfrom Mac OS X, applications developed for OS X do not run on iOS.Therefore, while many applications are available for mobile operatingsystems such as Android and iOS, many other common applications fordesktop operating systems such as Linux and Mac OS X are not availableon the mobile platforms.

Accordingly, smartphones are typically suited for a limited set of userexperiences and provide applications designed primarily for the mobileenvironment. In particular, smartphones do not provide a suitabledesktop user experience, nor do they run most common desktopapplications. As a result, many users carry and use multiple computingdevices including a smartphone, laptop, and/or tablet computer. In thisinstance, each device has its own CPU, memory, file storage, andoperating system.

Connectivity and file sharing between smartphones and other computingdevices involves linking one device (e.g., smartphone, running a mobileOS) to a second, wholly disparate device (e.g., notebook, desktop, ortablet running a desktop OS), through a wireless or wired connection.Information is shared across devices by synchronizing data betweenapplications running separately on each device. This process, typicallycalled “synching,” is cumbersome and generally requires activemanagement by the user.

SUMMARY

Embodiments of the present invention are directed to providing themobile computing experience of a smartphone and the appropriate userexperience of a secondary terminal environment in a single mobilecomputing device. A secondary terminal environment may be somecombination of visual rendering devices (e.g., monitor or display),input devices (e.g., mouse, touch pad, touch-screen, keyboard, etc.),and other computing peripherals (e.g., HDD, optical disc drive, memorystick, camera, printer, etc.) connected to the computing device by awired (e.g., USB, Firewire, Thunderbolt, etc.) or wireless (e.g.,Bluetooth, WiFi, etc.) connection. In embodiments, a mobile operatingsystem associated with the user experience of the mobile environment anda desktop operating system associated with the user experience of thesecondary terminal environment are run concurrently and independently ona shared kernel.

According to one aspect consistent with various embodiments, a firstoperating system and a second operating system run concurrently on ashared kernel of a mobile computing device. A first application and asecond application are in active concurrent execution within the firstoperating system, the first application displayed within a first userenvironment associated with the first operating system and the secondapplication displayed within a second user environment associated with asecond operating system. The first operating system maintainsapplication graphics for the second application by rendering a graphicsframe for the second application through a first virtual display of anextended rendering context. One method of facilitating user interactionfor the mobile computing device includes receiving a first user inputevent in a first operating system, establishing an extended input queueof the first operating system having a first motion space and a secondmotion space, the second motion space associated with the first virtualdisplay, receiving the first user input event at a first virtual inputdevice from the first console application of the second operatingsystem, mapping the first virtual input device to the second motionspace of the extended input queue of the first operating system, andpassing the first user input event to the second application from themapped first virtual input device.

According to other aspects consistent with various embodiments, a thirdapplication is in active concurrent execution with the first and secondapplications within the first operating system, the first operatingsystem rendering application graphics for the third application to asecond virtual display, wherein the first virtual display and the secondvirtual display are not adjacent to each other in the extended inputqueue. The first virtual display and the second virtual display may beoffset within the extended input queue by a virtual display offset thatis substantially larger than a resolution of virtual displays of thefirst operating system. A second console application of the secondoperating system may display application graphics for the thirdapplication rendered by the first operating system to the second virtualdisplay.

According to other aspects consistent with various embodiments, themethod may include receiving a second user input command at a secondvirtual input device from the second console application, mapping thesecond virtual device to a third motion space of the extended inputqueue of the first operating system, and passing the second user inputcommand to the third application from the mapped second virtual inputdevice. The first console application may translate the first user inputevent from a first input event protocol of the window system of thesecond operating system to a second input event protocol of the firstoperating system. The first operating system may be a mobile operatingsystem and the second operating system may be a desktop operatingsystem.

According to other aspects consistent with various embodiments, a mobilecomputing device includes a first application and a second applicationin active concurrent execution within a first operating system. Themobile computing device includes a first input device that receives afirst input event from an input element of a first user environment, thefirst user environment associated with the first operating system, and asecond input device that receives a second input event from an inputelement of a second user environment, the second user environmentassociated with a second operating system, the second operating systemrunning concurrently with the first operating system on a shared kernel.The mobile computing device further includes a console applicationrunning in the second operating system that receives the second inputevent and passes a third input event based on the second input event toa virtual input device accessible by the first operating system, and anextended input queue of the first operating system, the extended inputqueue including a first motion space associated with a local display ofthe first operating system and a second motion space associated with afirst virtual display of the first operating system, the first virtualdisplay associated with the second application, wherein the firstoperating system maps the virtual input device to the second motionspace. A graphics server of the second operating system may be anX-windows type graphics server and the second application may use agraphics library of the first operating system that is incompatible withthe X-windows type graphics server of the second operating system.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are illustrated in referencedfigures of the drawings, in which like numbers refer to like elementsthroughout the description of the figures.

FIG. 1 illustrates a computing environment that provides multiple usercomputing experiences, according to various embodiments.

FIG. 2 illustrates an exemplary system architecture for a mobilecomputing device, according to various embodiments.

FIG. 3 illustrates an operating system architecture for a computingenvironment, according to various embodiments.

FIG. 4 illustrates an exemplary computing environment employing variousaspects of embodiments.

FIG. 5 illustrates aspects of an operating system architecture for acomputing environment, according to various embodiments.

FIG. 6 illustrates an exemplary boot procedure that may be used toconfigure an operating system architecture of a mobile computing devicein more detail, according to various embodiments.

FIG. 7 illustrates an operating system architecture configuration forproviding cross-environment rendering of applications and/or userinteraction spaces, according to various embodiments.

FIG. 8 illustrates a computing environment with multiple userenvironments, according to various embodiments.

FIG. 9 illustrates aspects of cross-environment remote rendering,according to various embodiments.

FIG. 10 shows a flow diagram of an illustrative method forcross-environment remote rendering in a non-extended rendering context,according to various embodiments.

FIG. 11 illustrates a registration and drawing process flow forcross-environment remote rendering, according to various embodiments.

FIG. 12 shows a flow diagram of another illustrative method forcross-environment rendering in a non-extended rendering context,according to various embodiments.

FIG. 13 illustrates operating system architecture configuration 300 bfor providing user interaction support to cross-environmentapplications, according to various embodiments.

FIG. 14 illustrates aspects of user interaction support forcross-environment applications rendered using a non-extended graphicscontext, according to various embodiments.

FIG. 15 illustrates aspects of concurrent user interface support acrossmultiple OSs using extended rendering contexts, according to variousembodiments.

FIG. 16 shows a flow diagram of an illustrative method forcross-environment remote rendering in an extended rendering context,according to various embodiments.

FIG. 17 shows a flow diagram of another illustrative method forcross-environment rendering in an extended rendering context, accordingto various embodiments.

FIG. 18 a illustrates a user environment that may be employed incross-environment rendering, in an extended rendering context, accordingto various embodiments.

FIG. 18 b illustrates an extended input queue that may be employed incross-environment rendering, in an extended rendering context, accordingto various embodiments.

FIG. 19 illustrates a method for receiving input events that may beemployed in cross-environment rendering, in an extended renderingcontext, according to various embodiments.

FIG. 20 shows a flow diagram of an illustrative method forcross-environment rendering to provide a mirrored context, according tovarious embodiments.

FIG. 21 shows a flow diagram 2100 of another illustrative method forcross-environment rendering to provide a mirrored context, according tovarious embodiments.

FIG. 22 illustrates aspects of cross-environment redirection, accordingto various embodiments.

FIG. 23 illustrates a flow diagram of an illustrative method that may beemployed to perform aspects of cross-environment redirection, accordingto various embodiments.

FIG. 24 illustrates a flow diagram of another illustrative method thatmay be employed to perform aspects of cross-environment redirection,according to various embodiments.

DETAILED DESCRIPTION

Mobile telephony devices, (i.e., smartphones, handsets, mobile stations,portable communication devices, etc.) that include computingcapabilities are increasing in popularity. Many of these smartphonesinclude a mobile operating system (“OS”) running on a mobile processor.While mobile processors and mobile OS's have increased the capabilitiesof these devices, smartphones have not tended to replace personalcomputer (“PC”) environments (i.e., Windows, Mac OS X, Linux) such asdesktop or notebook computers at least because of the limited userexperience provided. In particular, the user interface device(s) foundon smartphones are typically tailored to the mobile environment. Forexample, smartphones typically use a small thumb-style QWERTY keyboard,touch-screen display, click-wheel, and/or scroll-wheel as user interfacedevices. Mobile OSs, as well as applications (i.e., “Apps”) developedfor mobile OSs, are typically designed for the constraints of the mobileenvironment including a mobile processor and the user interfacedevice(s) present on mobile devices. Therefore, many applications thathave been developed for PC operating systems are not available formobile OSs (i.e., are not compiled for and do not run on mobile OSs). Inaddition, for some tasks such as typing or editing documents, afull-size keyboard and large display are easier to use than the userinterface components typically found on a smartphone.

Accordingly, users typically use separate computing devices for eachcomputing experience, including a smartphone, tablet computer, laptopcomputer, and/or desktop computer. In this instance, each device has itsown CPU, memory, file storage, and OS. Connectivity and file sharingbetween smartphones and other devices involves linking one device (e.g.,smartphone, running a mobile OS) to a second, wholly disparate device(e.g., notebook, desktop, or tablet running a desktop OS), through awireless or wired connection. Information is shared across devices bysynchronizing data between applications running separately on eachdevice. This process, typically called “synching,” is cumbersome andgenerally requires active management by the user.

FIG. 1 illustrates a computing environment 100 that provides multipleuser computing experiences with a mobile device that includes multipleoperating systems associated with separate user interaction spaces(i.e., user environments), according to various embodiments. A firstuser interaction space 115 of computing environment 100 includesdisplay(s) 116 and I/O devices 118 of mobile computing device 110. Whenmobile computing device 110 is operated as a stand-alone mobile device,mobile OS 130 presents a typical mobile computing user experiencethrough user interaction space 115. The mobile computing experienceprovided by mobile OS 130 typically includes mobile telephonycapabilities and a graphical user interface (“GUI”) suited to userinteraction space 115 including display(s) 116 and I/O device(s) 118.For example, display(s) 116 may be a touch-screen display(s) andapplication programs (i.e., “Apps”) running on mobile OS 130 may becontrolled primarily through a gesture-based GUI of mobile OS 130 usingtouch-screen display(s) 116.

In computing environment 100, mobile computing device 110 may be dockedwith secondary terminal environment 140 that includes I/O devices 144,146, and/or 148. In embodiments, mobile computing device 110 is dockedwith secondary terminal environment 140 by connecting port 120 of mobilecomputing device 110 to port 142 of secondary terminal environment 140.In this instance, secondary terminal environment 140 presents a seconduser interaction space of computing environment 100. In some instances,the second user interaction space may be more suited to a desktopcomputing experience. In these instances, desktop OS 160 can beassociated with secondary terminal environment 140 to provide the fullcapabilities of a notebook, tablet, or desktop computer environmentthrough the second user interaction space.

In embodiments, mobile OS 130 and desktop OS 160 run concurrently on ashared kernel on a processor of mobile computing device 110. Concurrentexecution of a mobile OS and a desktop OS on a shared kernel isdescribed in more detail in U.S. patent application Ser. No. 13/217,108,filed Aug. 24, 2011, entitled “MULTI-OPERATING SYSTEM,” hereinincorporated by reference. In this way, a single mobile computing devicecan provide a mobile computing experience through a first userinteraction space and a desktop computing experience through a seconduser interaction space. While the ability to carry one mobile devicethat can execute multiple operating systems concurrently throughseparate user interaction spaces solves a number of problems for a user,each user interaction space (through the concurrently running mobile OSand desktop OS) generally provides a separate set of availableapplications and user functionality.

Embodiments of the invention are directed to facilitating apparentexecution of an application running in a first OS (e.g., mobile OS 130)within a second OS (e.g., desktop OS 160), where the first and second OSare running concurrently on a shared kernel. Notably, providing a userwith input (e.g., input device) and output (e.g., display, audio, etc.)support in a second OS for applications compiled for and running in afirst (e.g., incompatible) OS involves addressing a number of issues.Additional issues can arise when handling display and interactivity ofmultiple applications running concurrently.

Consider, for example, that a first and second application are bothcompiled for the first OS and are running concurrently on the first OS.However, a user desires to view graphical output of the firstapplication and to interact with that first application throughinput/output devices associated with the first OS (e.g., using atouchscreen display of a mobile computing environment), and to viewgraphical output of the second application and to interact with thatsecond application through input/output devices associated with thesecond OS (e.g., using a display, keyboard, and mouse of a desktopcomputing environment). Handling this scenario involves concurrenthandling of graphics in multiple display environments and concurrentprocessing of multiple input/output streams for separate applicationsall on separate (e.g., incompatible) operating systems.

Accordingly, embodiments provide various novel techniques for accessingapplications of a first OS within a user interaction space of a secondOS, displaying applications running in the first OS within the userinteraction space of the second OS, and handling user interaction withthose applications through the user interaction space of the second OS.Embodiments include a console application of the second OS that supportsvarious display and user interaction features of cross-environmentapplications.

One set of embodiments provides techniques for concurrent user interfacesupport across multiple-OS computing environments using a so-called“non-extended” rendering context. Another set of embodiments providestechniques for concurrent user interface support across multiple-OScomputing environments using a so-called “extended” rendering context.Yet another set of embodiments provides techniques for concurrent userinterface support across multiple OSs using a so-called “mirrored”context. Yet another set of embodiments provides access from the userinteraction space of the second OS to applications available on thefirst OS. Each of these sets of embodiments will be described more fullybelow.

As described above, computing environment 100 provides multiple usercomputing experiences through multiple user interaction spacesassociated with a mobile device running multiple operating systemsconcurrently. Specifically, because mobile computing device 110 includesmultiple OSs, where each OS is suited to a particular computingenvironment, mobile computing device 110 may be adapted with externalI/O devices to provide a broad range of user experiences with a singlemobile computing device. For example, a user may have a mobile computingdevice 110 and a secondary terminal environment 140 that includes akeyboard, display, and/or pointing device(s) in a laptop-type enclosure.When mobile computing device 110 is docked with this laptop-likesecondary terminal environment, the full capabilities of desktop OS 160are available through the secondary terminal environment 140.

FIG. 2 illustrates an exemplary hardware system architecture for mobilecomputing device 110, according to various embodiments. Mobile computingdevice hardware 112 includes mobile processor 114 that includes one ormore CPU cores 204 and external display interface 220. Generally, mobilecomputing device hardware 112 also includes I/O devices 118, memory 206,storage devices 208, touch-screen display controller 210 connected totouch-screen display 116, power management IC 214 connected to battery216, cellular modem 218, communication devices 222, and/or other devices224 that are connected to processor 114 through various communicationsignals and interfaces. I/O devices 118 generally includes buttons andother user interface components that may be employed in mobile computingdevice 110. For example, I/O devices 118 may include a set of buttons,(e.g., back, menu, home, search, etc.), off-screen gesture area,click-wheel, scroll-wheel, QWERTY keyboard, etc. Other devices 224 mayinclude, for example, GPS devices, LAN connectivity, microphones,speakers, cameras, accelerometers, and/or MS/MMC/SD/SDIO cardinterfaces. External display interface 220 may be any suitable displayinterface (e.g., VGA, DVI, HDMI, etc.).

Processor 114 may be an ARM-based mobile processor. In embodiments,mobile processor 114 is a mobile ARM-based processor such as TexasInstruments OMAP3430, Marvell PXA320, Freescale iMX51, or QualcommQSD8650/8250. However, mobile processor 114 may be another suitableARM-based mobile processor or processor based on other processorarchitectures such as, for example, x86-based processor architectures orother RISC-based processor architectures.

While FIG. 2 illustrates one exemplary hardware implementation 112 formobile computing device 110, other architectures are contemplated aswithin the scope of the invention. For example, various componentsillustrated in FIG. 2 as external to mobile processor 114 may beintegrated into mobile processor 114. Optionally, external displayinterface 220, shown in FIG. 2 as integrated into mobile processor 114,may be external to mobile processor 114. Additionally, other computerarchitectures employing a system bus, discrete graphics processor,and/or other architectural variations are suitable for employing aspectsof the present invention.

FIG. 3 illustrates OS architecture 300 that may be employed to runmobile OS 130 and desktop OS 160 concurrently on mobile computing device110, according to various embodiments. As illustrated in FIG. 3, mobileOS 130 and desktop OS 160 are independent operating systems.Specifically, mobile OS 130 and desktop OS 160 may have independent andincompatible user libraries, graphics systems, and/or framework layers.Functions and instructions for OS architecture 300 may be stored ascomputer program code on a tangible computer readable medium of mobilecomputing device 110. For example, instructions for OS architecture 300may be stored in storage device(s) 208 of mobile computing devicehardware 112.

In OS architecture 300, mobile OS 130 and desktop OS 160 runconcurrently on shared kernel 320. This means that mobile OS 130 anddesktop OS 160 are running on shared kernel 320 at the same time.Specifically, mobile OS 130 and desktop OS 160 both interface to sharedkernel 320 through the same kernel interface 322, for example, by makingsystem calls to shared kernel 320. Shared kernel 320 manages taskscheduling for processes of both mobile OS 130 and desktop OS 160. Inthis regard, mobile OS 130 and desktop OS 160 are running independentlyand concurrently on shared kernel 320. In addition, shared kernel 320runs directly on mobile processor 114 of mobile computing devicehardware 112, as illustrated by hardware interface 312. Specifically,shared kernel 320 directly manages the computing resources of mobilecomputing device hardware 112 such as CPU scheduling, memory access, andI/O. In this regard, hardware resources are not virtualized, meaningthat mobile OS 130 and desktop OS 160 make system calls through kernelinterface 322 without virtualized memory or I/O access.

As illustrated in FIG. 3, mobile OS 130 has libraries layer 330,application framework layer 340, and application layer 350. In mobile OS130, applications 352 and 354 run in application layer 350 supported byapplication framework layer 340 of mobile OS 130. Application frameworklayer 340 includes manager(s) 342 and service(s) 344 that are used byapplications running on mobile OS 130. For example, applicationframework layer 340 may include a window manager, activity manager,package manager, resource manager, telephony manager, gesturecontroller, and/or other managers and services for the mobileenvironment. Application framework layer 340 may include a mobileapplication runtime environment that executes applications developed formobile OS 130. The mobile application runtime environment may beoptimized for mobile computing resources such as lower processing powerand/or limited memory space. The mobile application runtime environmentmay rely on the kernel for process isolation, memory management, andthreading support. Libraries layer 330 includes user libraries 332 thatimplement common functions such as I/O and string manipulation, graphicsfunctions, database capabilities, communication capabilities, and/orother functions and capabilities.

As illustrated in FIG. 3, desktop OS 160 has libraries layer 360,framework layer 370, and application layer 380. In desktop OS 160,applications 382 and 384 run in application layer 380 supported byapplication framework layer 370 of desktop OS 160. Application frameworklayer 370 includes manager(s) 372 and service(s) 374 that are used byapplications running on desktop OS 160. For example, applicationframework layer 370 may include a window manager, activity manager,package manager, resource manager, and/or other managers and servicescommon to a desktop environment. Libraries layer 360 may include userlibraries 362 that implement common functions such as I/O and stringmanipulation, graphics functions, database capabilities, communicationcapabilities, and/or other functions and capabilities.

In various embodiments of the present disclosure, desktop OS 160 runs ina separate execution environment from mobile OS 130. For example, mobileOS 130 may run in a root execution environment and desktop OS 160 mayrun in a secondary execution environment established under the rootexecution environment. Processes and applications running on mobile OS130 access user libraries 332, manager(s) 342 and service(s) 344 in theroot execution environment. Processes and applications running ondesktop OS 160 access user libraries 362, manager(s) 372 and service(s)374 in the secondary execution environment.

In embodiments, mobile OS 130 and desktop 160 are independent operatingsystems with incompatible user libraries, graphics systems, and/orapplication frameworks. Therefore, applications developed for mobile OS130 may not run directly on desktop OS 160, and applications developedfor desktop OS 160 may not run directly on mobile OS 130. For example,application 352, running in application layer 350 of mobile OS 130, maybe incompatible with desktop OS 160, meaning that application 352 couldnot run on desktop OS 160. Specifically, application 352 may depend onmanager(s) 342, service(s) 344, and/or libraries 332 of mobile OS 130that are either not available or not compatible with manager(s) 372,service(s) 374, and/or libraries 362 of desktop OS 160.

As a result, mobile OS 130 and desktop OS 160 may have different sets ofavailable applications. In this regard, mobile OS 130 and desktop OS 160of OS architecture 300 provide separate user experiences throughseparate sets of applications accessible through separate userinteraction spaces. The user may access the applications available on(i.e., compiled for and loaded within the execution environment of)mobile OS 130 through a first user interaction space associated withmobile OS 130, and the applications available on desktop OS 160 througha second user interaction space associated with desktop OS 160.

As described above, mobile operating systems typically do not use thesame graphics environment as desktop operating systems. Graphicsenvironments for desktop OS's were designed for flexibility and highperformance. For example, the X-window system, used by some desktop OSs,provides platform and network independence at the expense of greaterprocessing and system resources. In contrast, graphics environments formobile OSs are designed more for efficiency and the specific user inputdevices of a mobile computing environment and less for flexibility.Because the graphics environments of mobile and desktop OS's are oftendifferent, an application running on a mobile OS may not be re-directedto display within a user space of a desktop OS by re-directing thegraphics information from the graphics server of the mobile OS to thegraphics server of the desktop OS.

The most widely adopted mobile OS is Google's Android. While Android isbased on Linux, it includes modifications to the kernel and other OSlayers for the mobile environment and mobile processors. In particular,while the Linux kernel is designed for a PC (i.e., x86) CPUarchitecture, the Android kernel is modified for ARM-based mobileprocessors. Android device drivers are also particularly tailored fordevices typically present in a mobile hardware architecture includingtouch-screens, mobile connectivity (GSM/EDGE, CDMA, Wi-Fi, etc.),battery management, GPS, accelerometers, and camera modules, among otherdevices. In addition, Android does not have a native X Window System nordoes it support the full set of standard GNU libraries, and this makesit difficult to port existing GNU/Linux applications or libraries toAndroid.

Apple's iOS operating system (run on the iPhone) and Microsoft's WindowsPhone 7 are similarly modified for the mobile environment and mobilehardware architecture. For example, while iOS is derived from the Mac OSX desktop OS, common Mac OS X applications do not run natively on iOS.Specifically, iOS applications are developed through a standarddeveloper's kit (“SDK”) to run within the “Cocoa Touch” runtimeenvironment of iOS, which provides basic application infrastructure andsupport for key iOS features such as touch-based input, pushnotifications, and system services. Therefore, applications written forMac OS X do not run on iOS without porting. In addition, it may bedifficult to port Mac OS X applications to iOS because of differencesbetween user libraries and/or application framework layers of the twoOSs, and/or differences in system resources of the mobile and desktophardware.

In one embodiment consistent with OS architecture 300, an Android mobileOS and a full Linux OS run independently and concurrently on a modifiedAndroid kernel. In this embodiment, the Android OS may be a modifiedAndroid distribution while the Linux OS (“Hydroid”) may be a modifiedDebian Linux desktop OS. FIGS. 4-6 illustrate Android mobile OS 430,Android kernel 520, and Hydroid OS 660 that may be employed in OSarchitecture 300 in more detail, according to various embodiments.

As illustrated in FIG. 4, Android OS 430 includes a set of C/C++libraries in libraries layer 432 that are accessed through applicationframework layer 440. Libraries layer 432 includes the “bionic” system Clibrary 439 that was developed specifically for Android to be smallerand faster than the “glibc” Linux C-library. Libraries layer 432 alsoincludes inter-process communication (“IPC”) library 436, which includesthe base classes for the “Binder” IPC mechanism of the Android OS.Binder was developed specifically for Android to allow communicationbetween processes and services. Other libraries shown in libraries layer432 in FIG. 4 include media libraries 435 that support recording andplayback of media formats, surface manager 434 that managers access tothe display subsystem and composites graphic layers from multipleapplications, 2D and 3D graphics engines 438, and lightweight relationaldatabase engine 437. Other libraries that may be included in librarieslayer 432 but are not pictured in FIG. 4 include bitmap and vector fontrendering libraries, utilities libraries, browser tools (i.e., WebKit,etc.), and/or secure communication libraries (i.e., SSL, etc.).

Application framework layer 440 of Android OS 430 provides a developmentplatform that allows developers to use components of the devicehardware, access location information, run background services, setalarms, add notifications to the status bar, etc. Framework layer 440also allows applications to publish their capabilities and make use ofthe published capabilities of other applications. Components ofapplication framework layer 440 of Android mobile OS 430 includeactivity manager 441, resource manager 442, window manager 443, dockmanager 444, hardware and system services 445, desktop monitor service446, multi-display manager 447, and remote communication service 448.Other components that may be included in framework layer 440 of Androidmobile OS 430 include a view system, telephony manager, package manager,location manager, and/or notification manager, among other managers andservices.

Applications running on Android OS 430 run within the Dalvik virtualmachine 431 in the Android runtime environment 433 on top of the Androidobject-oriented application framework. Dalvik virtual machine 431 is aregister-based virtual machine, and runs a compact executable formatthat is designed to reduce memory usage and processing requirements.Applications running on Android OS 430 include home screen 451, emailapplication 452, phone application 453, browser application 454, and/orother application(s) (“App(s)”) 455.

The Android OS graphics system uses a client/server model. A surfacemanager (“SurfaceFlinger”) is the graphics server and applications arethe clients. SurfaceFlinger maintains a list of display ID's and keepstrack of assigning applications to display ID's. In one embodiment,mobile computing device 110 has multiple touch screen displays 116. Inthis embodiment, display ID 0 is associated with one of the touch screendisplays 116 and display ID 1 is associated with the other touch screendisplay 116. Display ID 2 is associated with both touch screen displays116 (i.e., the application is displayed on both displays at the sametime). Display ID's greater than 2 are virtual displays, meaning thatthey are not associated with a display physically present on mobilecomputing device hardware 112.

Graphics information for Android applications includes windows, views,and canvasses. Each window, view, and/or canvas is implemented with anunderlying surface object. Surface objects are double-buffered (frontand back buffers) and synchronized across processes for drawing.SurfaceFlinger maintains all surfaces in a shared memory pool whichallows all processes within Android to access and draw into them withoutexpensive copy operations and without using a server-side drawingprotocol such as X-Windows. Applications always draw into the backbuffer while SurfaceFlinger reads from the front buffer. SurfaceFlingercreates each surface object, maintains all surface objects, and alsomaintains a list of surface objects for each application. When theapplication finishes drawing in the back buffer, it posts an event toSurfaceFlinger, which swaps the back buffer to the front and queues thetask of rendering the surface information to the frame buffer.

SurfaceFlinger monitors all window change events. When one or morewindow change events occur, SurfaceFlinger renders the surfaceinformation to the frame buffer for one or more displays. Renderingincludes compositing the surfaces, i.e., composing the final image framebased on dimensions, transparency, z-order, and visibility of thesurfaces. Rendering may also include hardware acceleration (e.g., OpenGL2D and/or 3D interface for graphics processing hardware). SurfaceFlingerloops over all surface objects and renders their front buffers to theframe buffer in their Z order.

FIG. 5 illustrates modified Android kernel 520 in more detail, accordingto various embodiments. Modified Android kernel 520 includestouch-screen display driver 521, camera driver(s) 522, Bluetoothdriver(s) 523, shared memory allocator 524, IPC driver(s) 525, USBdriver(s) 526, WiFi driver(s) 527, I/O device driver(s) 528, and/orpower management module 530. I/O device driver(s) 528 includes devicedrivers for external I/O devices, including devices that may beconnected to mobile computing device 110 through port 120. ModifiedAndroid kernel 520 may include other drivers and functional blocksincluding a low memory killer, kernel debugger, logging capability,and/or other hardware device drivers.

FIG. 6 illustrates Hydroid OS 660 in more detail, according to variousembodiments. Hydroid is a full Linux OS that is capable of runningalmost any application developed for standard Linux distributions. Inparticular, libraries layer 662 of Hydroid OS 660 includes Linuxlibraries that support networking, graphics processing, databasemanagement, and other common program functions. For example, userlibraries 662 may include the “glibc” Linux C library 664, Linuxgraphics libraries 662 (e.g., GTK, OpenGL, etc.), Linux utilitieslibraries 661, Linux database libraries, and/or other Linux userlibraries. Applications run on Hydroid within an X-Windows Linuxgraphical environment using X-Server 674, window manager 673, and/ordesktop environment 672. Illustrated applications include word processor681, email application 682, spreadsheet application 683, browser 684,and other application(s) 685.

The Linux OS graphics system is based on the X-windows (or “X11”)graphics system. X-windows is a platform-independent, networked graphicsframework. X-windows uses a client/server model where the X-server isthe graphics server and applications are the clients. The X-servercontrols input/output hardware associated with the Linux OS such asdisplays, touch-screen displays, keyboards, pointing device(s), etc. Inthis regard, X-windows provides a server-side drawing graphicsarchitecture, i.e., the X-server maintains the content for drawablesincluding windows and pixmaps. X-clients communicate with the X-serverby exchanging data packets that describe drawing operations over acommunication channel. X-clients access the X communication protocolthrough a library of standard routines (the “Xlib”). For example, anX-client may send a request to the X-server to draw a rectangle in theclient window. The X-server sends input events to the X-clients, forexample, keyboard or pointing device input, and/or window movement orresizing. Input events are relative to client windows. For example, ifthe user clicks when the pointer is within a window, the X-server sendsa packet that includes the input event to the X-client associated withthe window that includes the action and positioning of the eventrelative to the window.

Because of the differences in operating system frameworks, graphicssystems, and/or libraries, applications written for Android do notgenerally run on Hydroid OS 660 and applications written for standardLinux distributions do not generally run on Android OS 430. In thisregard, applications for Android OS 430 and Hydroid OS 660 are notbytecode compatible, meaning compiled and executable programs for one donot run on the other.

In one embodiment, Hydroid OS 660 includes components of across-environment communication framework that facilitates communicationwith Android OS 430 through shared kernel 520. These components includeIPC library 663 that includes the base classes for the Binder IPCmechanism of the Android OS and remote communications service 671.

In one embodiment, Hydroid OS 660 is run within a chrooted (created withthe ‘chroot’ command) secondary execution environment created within theAndroid root environment. Processes and applications within Hydroid OS660 are run within the secondary execution environment such that theapparent root directory seen by these processes and applications is theroot directory of the secondary execution environment. In this way,Hydroid OS 660 can run programs written for standard Linux distributionswithout modification because Linux user libraries 662 are available toprocesses running on Hydroid OS 660 in the chrooted secondary executionenvironment.

Referring back to FIG. 3, mobile OS 130 and desktop 160 are in activeconcurrent execution on shared kernel 320 on a mobile device. Mobile OS130 and desktop OS 160 may be incompatible with regard to userlibraries, graphics systems, and/or application frameworks. Therefore,mobile OS 130 and desktop OS 160 have different sets of availableapplications, meaning that at least some applications available onmobile OS 130 are not available on desktop OS 160 and vice-versa.Accordingly, mobile OS 130 and desktop OS 160 of OS architecture 300provide separate user experiences through different sets of applicationsaccessible through separate user interaction spaces. The user may accessthe applications available on (i.e., compiled for and loaded within theexecution environment of) mobile OS 130 through the user interactionspace associated with mobile OS 130, and the applications available ondesktop OS 160 through the user interaction space associated withdesktop OS 160.

Embodiments of the present invention extend the functionality of OSarchitecture 300 to provide a more seamless computing experience in amulti-OS computing environment. Embodiments include cross-environmentrendering of applications and/or the user interaction space of a firstoperating system within a user interaction space of a second operatingsystem, even where the graphics environments of the first and secondoperating systems are not compatible. Embodiments further includeuser-interaction support of cross-environment applications, andaccessing applications of the first operating system from the userinteraction space of the second operating system. This functionalityenables, for example, mobile OS applications, running on mobile OS 130,to be displayed and interacted with through a user interaction spaceassociated with desktop OS 160. For example, while a user is interactingwith desktop OS 160 through a user interaction space associated withdesktop OS 160, the user may wish to have access to a particularapplication of mobile OS 130 that is not available for (i.e., is notcompiled for and does not run on) desktop OS 160. Using variousembodiments disclosed below, the user may access, display, and interactwith an application compiled for and running on mobile OS 130 throughthe user interaction space associated with desktop OS 160. Notably, theembodiments provide cross-environment interaction support with anyapplication of the mobile OS, meaning that mobile OS applications do notneed to be modified to include specific cross-environment support to usethe embodiments below.

To provide seamless cross-environment user interaction support for anapplication and/or the user interaction space of a first OS (e.g.,mobile OS 130) from within a user interaction space of a second OS(e.g., desktop OS 160), it may be desirable for graphics data of theapplication and/or the user interaction space (i.e., graphics context oractive display) to be rendered for display in the user interaction spaceof the second OS in real-time. Real-time (or instant) rendering, in thiscontext, means that graphics data of the application is rendered to theuser interaction space of the second OS fast enough that the user caninteract with the application without a noticeable or substantialreduction in application performance due to delays associated withtransferring the graphics information. In this regard, the techniquesfor real-time (or instant) cross-environment rendering, as describedherein, provide for rapid transfer of graphics information, for example,with a limited number of frame delays or other delays associated withcopying or transferring the graphics information from the first OS tothe second OS. However, it does not mean that the graphics transfer doesnot take any time, and the cross-environment rendering techniquesdisclosed herein may be considered instant or in real-time, even thougha finite time period passes before the graphics information is displayedon the user interaction space of the second OS.

To achieve cross-environment rendering of applications, a potentiallylarge amount of graphics data may be passed from the application runningin the first operating system to a graphics system of the secondoperating system. Existing mechanisms are not capable of transferringthe required graphics data without potentially affecting the displayupdate or frame rate. For example, transferring graphics data directlywould not be practical within the constraints of a mobile computingenvironment. Compression techniques could be used to reduce the totalamount of data to be transferred, but at the cost of increasedprocessing requirements to compress and de-compress the information.Remote desktop-like systems could be used to pass vector graphics orgraphics update information, however, these are also typically slow whentransferring large amounts of graphics data or for rapidly changinggraphics content.

FIG. 7 illustrates OS architecture configuration 300 a, which may beemployed to provide cross-environment rendering of an application and/orthe user interaction space of a first OS within a user interaction spaceof a second OS, where the first and second OS are running concurrentlyon a shared kernel and are associated with separate user interactionspaces that include separate display devices, according to variousembodiments. Components of OS architecture configuration 300 a allowapplications running on mobile OS 130 and/or the graphics context ofmobile OS 130 to be rendered within the user interaction space ofdesktop OS 160, where OS 130 and desktop OS 160 run concurrently onshared kernel 320. In various embodiments, an application is displayedin a console window in the user interaction space of desktop OS 160through console application 782 of desktop OS 160. In oneimplementation, console application 782 is an X-windows type applicationthat is displayed within the user interaction space of desktop OS 160through an X-windows type graphics system of desktop OS 160.

In OS architecture configuration 300 a, mobile OS 130 includes graphicsserver 734 that allocates and manages surface information (e.g.,surfaces 726, 727, and 728) for applications (e.g., application 752 and754) of mobile OS 130. In one embodiment, graphics server 734 allocatesmemory for graphics surfaces using anonymous shared memory (i.e., namedmemory blocks shared between processes that the kernel is allowed tofree). Graphics server 734 also allocates and tracks displays of mobileOS 130, including displays that are integrated within mobile computingdevice hardware 112 (i.e., local displays) and so-called virtualdisplays through which applications of mobile OS 130 may be displayedremotely (i.e., within a user interaction space of another OS). MobileOS 130 may also include a window system for displaying multipleapplications at the same time on a display screen associated with mobileOS 130. In one embodiment, mobile OS 130 includes a service thatprovides a remote communication channel for access to components ofmobile OS 130 from desktop OS 160.

OS architecture configuration 300 a includes a first application 752that is running on mobile OS 130 and displayed within a first userinteraction space associated with mobile OS 130. OS architectureconfiguration 300 a includes a second application 754 that is alsorunning on mobile OS 130, but is displayed within a second userinteraction space associated with desktop OS 160 throughcross-environment rendering according to embodiments described below. Inone embodiment, application 754 is displayed within a console window ofthe second user interaction space through console application 782running on desktop OS 160.

Generally, applications of mobile OS 130 instantiate views through whichthe user interacts with the application. For example, an application mayhave a single view that makes up the application window. Within views,applications instantiate drawing objects for specific areas of theapplication interface. The drawing objects may be called canvases ordrawing layers and are the objects through which the application drawsgraphics information. When an application instantiates a canvas orlayer, graphics server 734 allocates memory for the surface informationassociated with the canvas or layer and returns the drawing object tothe application, which the application then uses to draw graphicsinformation for the surfaces of the canvas or layer. Graphics server 734then monitors the surface information and renders the surfaceinformation into the frame buffer (e.g., frame buffer 716) when thesurface information is updated. Typically, the user also interacts withthe application through the view objects. The view objects include eventlisteners that are called by the mobile OS input queue 736 when actionsare performed by the user on the view object.

As illustrated in OS architecture 300 a, surfaces 726, 727, and/or 728are allocated by graphics server 734 in shared memory 724. Shared memory724 is managed by shared kernel 320 and is accessible by all processesrunning on mobile OS 130 and desktop OS 160. As described above, sharedmemory 724 may be named shared memory. While named shared memory isaccessible by all processes running on shared kernel 320, otherprocesses cannot access regions of named shared memory by name.Accordingly, a file descriptor to regions of named shared memory must bepassed through an inter-process communication mechanism to pass areference to named shared memory across process boundaries. Sharedkernel 320 also includes IPC driver 725, which allows processes inmobile OS 130 and desktop OS 160 to communicate with one another acrossprocess boundaries. IPC driver 725 may be, for example, a Unix domainsocket driver, Android Binder driver, and/or network socket driver.

Desktop OS 160 of OS architecture configuration 300 a includes consoleapplication 782 and window system 774. Console application 782 iscompiled for and runs on desktop OS 160, and is displayed within aconsole window in the user interaction space associated with desktop OS160. Window system 774 may include a window manager, graphics server,and/or graphics device interface that provides the basis forrepresenting graphical objects and transmitting them to display devicesthrough desktop OS frame buffer 718. For example, window system 774 maydisplay console application 782 within the user interaction space ofdesktop OS 160 through desktop OS frame buffer 718. Window system 774also provides input events from the user environment of desktop OS 160associated with console application 782 to console application 782. Forexample, window system 774 may provide pointing device location orgesture information from the user interaction space associated withdesktop OS 160 to console application 782.

In FIG. 7, memory blocks including shared memory 724, mobile OS framebuffer 716, and desktop OS frame buffer 718 are shown as located withinshared kernel 320 for ease of illustration. However, these memory blocksare physically located in tangible memory storage elements of mobilecomputing device 110 and managed by shared kernel 320. For example,these memory blocks may be located in RAM on processor 114, and/or inRAM of memory device 206 of mobile computing device hardware 112illustrated in FIG. 2.

OS architecture configuration 300 a provides support forcross-environment rendering of applications of a first operating systemrunning on a shared kernel within a user interaction space associatedwith a second operating system, where the first and second operatingsystems run concurrently on the shared kernel. OS architectureconfiguration 300 a may be employed to provide support forcross-environment rendering in a computing environment that providesmultiple user computing experiences through multiple user interactionspaces. For example, OS architecture configuration 300 a may be used incomputing environment 100 as illustrated in FIG. 1.

FIG. 8 illustrates computing environment 800, according to variousembodiments. Computing environment 800 has a first user environment thatincludes touch-screen display(s) 116 and other I/O devices 118 of mobilecomputing device hardware 112. This user environment presents a firstuser interaction space through which the user interacts with mobile OS130. A second user environment of computing environment 800 includesdisplay monitor 844, keyboard 846, and/or pointing device 848. Userenvironment 840 may be connected to mobile computing device 110 throughdock connector 841 and dock cable 843. Dock connector 841 and dock cable843 may include a port that interfaces through dock interface 122 toport 120 of mobile computing device 110 as illustrated in FIG. 1. Inthis regard, user environment 840 provides a docked secondary terminalenvironment to mobile computing device 110. In computing environment800, desktop OS 160 may be associated with docked secondary terminalenvironment 840 such that the user can interact with desktop OS 160through the user interaction space provided by secondary terminalenvironment 840. While secondary terminal environment 840 is illustratedas a typical desktop-type computing environment, desktop OS 160 maypresent a second user interaction space through other types of computingenvironments, including laptop, tablet, and/or other types of computingenvironments.

OS architecture configuration 300 a may be used within computingenvironment 800 to provide cross-environment rendering of applicationsrunning on a first OS (i.e., mobile OS) and displayed within a userenvironment of a second OS (i.e., user environment 840 associated withdesktop OS 160). For example, application 754, running on mobile OS 130,may be displayed within console window 882 on the user interaction space880 of desktop OS 160. Other windows within user interaction space 880,including window 884 and 886, may be windows of other applicationsrunning on desktop OS 160. To the user, computing environment 800provides a seamless computing experience for application 754 becauseapplication 754 can be used within the user interaction space of desktopOS 160 as if it was running on desktop OS 160, while in fact application754 is running on mobile OS 130.

FIG. 8 illustrates a desktop-like secondary terminal environment 840. Inthis instance, the user may interact with an application and/or the userinteraction space of the mobile OS through console window 882 usingkeyboard 846 and mouse 848 (i.e., a primarily pointing device-based GUI)of secondary terminal environment 840. However, cross-environmentrendering of an application and/or mirroring of the mobile OS userinteraction space may be used with other secondary terminalenvironments. For example, desktop OS 160 could be associated with atablet-like secondary terminal environment that includes a touch-screendisplay. In this instance, the user may interact with across-environment application or mirrored user interaction space ofmobile OS 130 in much the same manner (i.e., a primarily gesture-basedGUI) in which the user typically interacts with the user interactionspace of mobile OS 130.

As discussed above, embodiments of the invention are directed toproviding concurrent user interface support across cross-environmentapplications and/or a mirrored user interaction space in a multiple-OScomputing environment. In one example, user interface support isprovided for cross-environment applications to allow an application,running on a first OS, to be displayed and interacted with through auser interaction space of a second OS, substantially as if runningnatively on the second operating system.

Non-Extended Rendering Context Embodiments

Some embodiments handle concurrent user interface support acrossmultiple OSs without extending the graphics rendering context of thefirst operating system. The first OS (e.g., the mobile OS, Android) istypically configured to define a single, active user interaction space.The user interaction space includes an active display (e.g., withassociated characteristics, like resolution) and one or more activeinput devices for allowing user interaction with the elements displayedon the active display. Accordingly, the first OS establishes a renderingcontext through which it can render surface information for applicationsthat are running for display to the active display.

As described above, however, novel techniques are described herein foreffectively fooling the first OS into concurrently handling multipleuser interaction spaces. Moreover, the techniques allow the multipleuser interaction spaces to be associated with different (e.g.,incompatible) operating systems on multiple computing environments. Someembodiments involve techniques for handling the display outputs throughcross-environment remote rendering. Other embodiments involve techniquesfor handling the user interaction in those contexts.

In cross-environment remote rendering, application graphics for anapplication running on the first OS and displayed within a computingenvironment associated with a second OS are rendered from within thesecond OS. In one embodiment, a console application, running on thesecond OS, accesses surface information for the application from sharedmemory and renders the application within a console window of thecomputing environment associated with the second OS.

Suppose that a calendar application and a word processing applicationare both compiled for and concurrently running on a first OS (e.g.,mobile OS 130) on a mobile device. A second OS (e.g., a non-mobile OS,like desktop OS 160) is running concurrently on the mobile device usinga shared kernel. A user has docked the mobile device with a second,desktop computing environment, and desires to interact with the wordprocessing application through that desktop computing environment. It isdesirable to handle the user interaction space of the desktop computingenvironment using the second OS of the mobile computing environment(i.e., the mobile device) in a way that is transparent to the user.

FIG. 9 illustrates aspects of cross-environment remote rendering,according to various embodiments. In application rendering diagram 900illustrated in FIG. 9, the first application 910 (e.g., calendarapplication) calculates updates for a first surface 912 within the firstOS. The first surface 912 is stored in a first memory location in ashared memory space by the first operating system. For example, thefirst memory location may be a region of named shared memory. The firstapplication 910 updates the back buffer 914 of the first surface 912.Similarly, the second application 930 (e.g., word processingapplication) calculates updates for a second surface 932 using the firstOS. The second surface 932 is stored in a second memory location in theshared memory space (e.g., a second region of named shared memory) bythe first OS.

The first OS determines when to initiate a rendering sequence. Forexample, the first OS may initiate a rendering sequence when surfaceinformation for surfaces 912 and/or 932 has changed. The first OS mayperform a single loop over all surfaces, including the first surface 912and second surface 932, determining whether surface informationassociated with particular applications has changed. In the renderingsequence, if surface information for surface 912 has changed, the firstOS swaps the front buffer 916 and back buffer 914, such that the surfaceinformation that was in the back buffer 914 is now in front buffer 916.The first operating system renders the first surface 912 into a thirdmemory location 920 to create the final image 918 to be displayed withina first user interaction space associated with the first OS.

If surface information for the second surface 932 has changed, the firstOS notifies the second OS that the surface information for the secondsurface 932 has changed. Specifically, the first OS sends a drawnotification to a console application of the second OS indicating thatsurface 932 has been updated through an inter-process communicationchannel. The draw notification may include a file descriptor to thefront image 936, and/or other characteristics of the shared memory spacefor front image 936 including buffer size, layer order, etc. The consoleapplication maps the file descriptor to its process space to obtain areference to the second memory location. Through the reference to thesecond memory location, the console application of the second OS readsdirectly the surface information from the front image buffer 936 andrenders the surface information for the front image 936 of the secondsurface 932 in a console window 940 within a second user interactionspace associated with the second OS. In this way, the consoleapplication can render the second application 930 in the console window940 in real-time without copying a graphics frame or surface informationacross processes. Instead, the console application reads the surfaceinformation directly by mapping the shared memory of the second surface932 to its own process space using the file descriptor passed throughinter-process communication.

FIG. 10 shows a flow diagram 1000 of an illustrative method forcross-environment remote rendering in a non-extended rendering context,according to various embodiments. Embodiments maintain display ofapplication graphics for a first application (e.g., the calendarapplication) and a second application (e.g., the word processingapplication), both compiled for and in active concurrent executionwithin a first operating system.

The method 1000 begins at block 1004 by calculating updates to surfacesof the first application using the first operating system. Calculatingupdates to the surfaces may involve using application data to determinewhich surfaces have changed and in what ways. For example, userinteraction may have caused some surfaces to change position, order(e.g., one layer may be partially in front of another layer, completelyhidden by another layer, etc.), size, color, texture, etc.

At block 1008, these updated surfaces of the first application arerendered using the first operating system to generate a first graphicsframe in a first memory location. For example, a rendering engine of thefirst OS previously established a rendering context associated with adisplay. A graphics frame can then be rendered by iterating through theupdated surface information to effectively draw a full or partial imageof visible portions of surfaces from the first application according tocharacteristics (e.g., resolution) of the display associated with therendering context. This graphics frame is rendered to a first memorylocation. The memory location can be frame buffer memory, shared memory,or any other useful memory location.

In some embodiments, at block 1012, the rendered first graphics frame isdisplayed from the first memory location to a display of a firstcomputing environment associated with the first operating system. Forexample, the first graphics frame is rendered into a back buffer portionof the frame buffer of the mobile device. Subsequently, the frame bufferflips (i.e., the back buffer portion becomes the front buffer portion)and the now-front buffer portion of the frame buffer is displayed to thedisplay of the mobile device.

At block 1016, updates are calculated to surfaces of the secondapplication using the first operating system. This may be performedsubstantially identically to the calculation of block 1004 for surfacesof the first application. Unlike with the updated surface data of thefirst application, however, the updated surface information of thesecond application is not rendered by the first OS. Rather, at block1020, the updated surfaces of the second application are stored in asecond memory location. The second memory location is a shared memorylocation accessible by both the first and second OSs, which are runningconcurrently on the shared kernel of the mobile device.

At block 1024, the updated surfaces of the second application arerendered using a console application of the second operating system togenerate a second graphics frame in a third memory location. Forexample, console application 782 of FIG. 7 may render updated surfacesof application 754 into frame buffer memory of the second OS (e.g.,associated with the display of the second computing environment). Insome embodiments, at block 1028, the second graphics frame is displayedfrom the third memory location to a display of a second computingenvironment associated with the second operating system. For example,the display driver of the desktop computing environment display accessesthe frame buffer memory to access and display the second graphics frame.In certain implementations, the console application also maintains aconsole interaction space, and the second graphics frame is renderedinto the console interaction space. For example, the console applicationrenders one or more windows on the desktop display, and the graphics ofthe second application are displayed in one of those windows.

In some embodiments, the method 1000 iterates through the blocks toconcurrently maintain the graphics environments for both applications.For example, the mobile OS calculates updates to the first application'sgraphics and renders those updates to mobile frame buffer memory; thenthe mobile OS calculates updates to the second application's graphicsand stores those updates to shared memory, from where the desktop OS'sconsole application renders those updates to desktop frame buffermemory; then the method 1000 repeats with the next set of updates.Notably, some implementations perform certain steps out of order and/orin parallel. For example, it may be possible to calculate updates tosurfaces for the first and second applications at substantially the sametime (i.e., perform blocks 1004 and 1016 substantially in parallel),though only a portion of those updated surfaces will be rendered locally(e.g., at block 1008) and the other portion of those updated surfaceswill be stored for remote rendering (e.g., at blocks 1020 and 1024).

In one embodiment, an Android mobile OS and a full Linux OS (e.g.,Hydroid) are running concurrently on a shared kernel on a mobile device.When an Android application is launched from Hydroid OS 660, a consoleapplication launches on Hydroid OS 660 and requests that Android OS 430launch the Android application and send draw notifications for theAndroid application to the console application. Android OS 430 launchesthe Android application, associates the application with a virtualdisplay ID and registers the console application to receive drawnotifications for the application. For example, the Android graphicsserver (i.e., SurfaceFlinger) may allocate an unused virtual display IDand associate the application with the virtual display ID through anapplication object list. SurfaceFlinger allocates memory for surfaceinformation of the Android application in shared memory and registersthe console application to receive draw notifications for the Androidapplication.

When SurfaceFlinger determines that the surface information is updated,SurfaceFlinger sends a draw notification to the console applicationthrough an inter-process communication channel. For example,SurfaceFlinger may send the draw notification to the console applicationthrough a unix domain socket, Binder interface, and/or network socket.The draw notification includes a file descriptor to the surfaceinformation. The file descriptor may be generated by SurfaceFlingerbased on a namespace for a named shared memory region of the surface.The draw notification may also include data format, buffer size, andsurface position information. The console application maps the filedescriptor to its process space and reads from the shared memorylocation through the mapped file descriptor. The console applicationthen renders the graphics frame for the application directly from thesurface information. The rendered graphics frame is then displayed bythe desktop OS graphics system (i.e., through the X-windows graphicssystem of Hydroid OS 660).

FIG. 11 illustrates a registration and drawing process flow 1100 forcross-environment remote rendering in more detail, according to variousembodiments. Initially, console application 1102, running on Hydroid OS660, requests through IPC channel 525 of shared kernel 520 at step 1106that an Android application be started and displayed within a userenvironment of Hydroid OS 660, or moved from a user environment ofAndroid OS 430 to a user environment of Hydroid OS 660. Android requestsfrom SurfaceFlinger 434 a virtual display ID for the application, startsthe application, and sets parameters for the virtual display at steps1108, 1110, and 1112. At step 1114, Android returns the display ID tothe console application through IPC channel 525. At step 1116, theapplication requests a new surface, which SurfaceFlinger 434 creates bycreating an instance of surface class 1104 at step 1118.

Console application 1102 instantiates a renderer object at step 1120,which renders Android surface information through the X-window system ofHydroid OS 660. At step 1122, console application 1102 registers aremotable interface of the renderer object 1124 with SurfaceFlinger 434to receive draw notifications for surface information for the Androidapplication. In one embodiment, the remotable interface of the rendererobject includes draw( ) and clear( ) methods that may be called throughIPC channel 525. At steps 1126 and 1128, SurfaceFlinger attaches the IPCobject to the surface such that the console application 1102 will benotified through the remotable interface when the surface informationhas been updated.

Steps 1130 and 1132 are part of the render loop of process flow 1100. Inthe render loop, SurfaceFlinger notifies console application 1102 thatthe surface information is updated and passes a file descriptor to thesurface information through IPC channel 525. For example, a draw methodof the console application renderer may be called and passed the filedescriptor to the surface. Console application 1102 maps the filedescriptor to its process space and accesses the referenced sharedmemory location to read the surface information and render the graphicsframe to be displayed on a display of a second computing environmentassociated with Hydroid OS 660.

FIG. 12 shows a flow diagram 1200 of another illustrative method forcross-environment rendering in a non-extended rendering context,according to various embodiments. As with the method 1000 of FIG. 10,embodiments maintain display of application graphics for a firstapplication and a second application, both compiled for and in activeconcurrent execution within a first operating system.

The method 1200 begins at block 1204 by establishing a first renderingcontext of the first operating system. The rendering context may beunique to the display with which it is associated. For example,implementations of the mobile OS can be associated with a mobile devicedisplay having a certain resolution. The rendering context may beestablished to match the resolution of the associated display, so thatgraphics will be appropriately rendered for that resolution. At block1208, the method 1200 calculates updates to surfaces of the firstapplication using the first operating system. As discussed above,calculating updates to the surfaces may involve using application datato determine which surfaces have changed and in what ways.

The updated surfaces of the first application are then rendered usingthe first operating system, at block 1212, to generate a first graphicsframe in a first memory location. Rendering typically involvesconverting graphical primitives into bits (e.g., a bitmap) that form agraphics frame for display. For example, surface information definesmathematically properties of each surface, including shape, size, color,layering order (i.e., which other surfaces it is in front or in backof), transparency, etc. The rendering engine of the OS can interpret thesurface data to determine, at each location (e.g., “X, Y” location) inthe rendering context, what bit to display as a function of renderingall the surface information (e.g., using iterative compositing,ray-tracing, and/or other techniques). The updated surfaces of the firstapplication can be rendered, using the first rendering context, into abitmap for storage into the first memory location (e.g., frame buffermemory, shared memory, or any other useful memory location).

In some embodiments, at block 1216, the first graphics frame isdisplayed from the first memory location to a display of a firstcomputing environment associated with the first operating system. Forexample, a display driver for the mobile device display accessesassociated frame buffer memory to display the bitmap generated in thefirst rendering context. At block 1220, the first rendering context isdisestablished (e.g., “torn down”).

At block 1224, a second rendering context of the first operating systemis established. In some implementations, the second rendering context isidentical to the first rendering context. However, the second renderingcontext may also be established according to characteristics of adisplay of the second computing environment (e.g., the desktop display).Updates to surfaces of the second application are calculated using thefirst operating system at block 1228. These updates are then rendered,at block 1232, in the second rendering context of the first operatingsystem to generate a second graphics frame in a second memory location.Notably, the second memory location is a shared memory locationaccessible by both the first operating system and the second operatingsystem, which are running concurrently on the shared kernel of themobile device.

In some embodiments, at block 1236, the second graphics frame isdisplayed from the second memory location to a display of a secondcomputing environment associated with the second operating system. It isworth noting that, unlike in FIG. 10, both applications' updatedgraphics frames are rendered using the first OS's rendering engine. Forexample, the console application of the second OS can access the second,shared memory location to directly retrieve a rendered bitmap fordisplay to the second computing environment's display. At block 1240,the second rendering context is disestablished.

In some embodiments, the method 1200 iterates through the blocks toconcurrently maintain the graphics environments for both applications.For example, the mobile OS iteratively establishes, uses, and tears downa rendering context for the first application; then establishes, uses,and tears down a rendering context for the second application. Usingthis technique, all the rendering can be performed by one OS (i.e., bythe rendering engine of the first OS), and there is no need to provideor use rendering functionality of the other OS. However, the techniqueinvolves additional overhead associated with repeatedly establishing andtearing down rendering contexts.

The above embodiments of cross-environment rendering describe how agraphics frame for an application running on a first OS may be displayedwithin a console window of a user interaction space of a second OS in amulti-OS computing environment using non-extended rendering contexts. Tosupport user interaction with such cross-environment applications,embodiments redirect input events from the user interaction space of thesecond OS to the first OS in such a way that cross-environmentapplications receive input events as if coming from the user interactionspace of the first OS (i.e., the application receives the input eventsthrough the same event handlers and/or views through which it wouldreceive user input if displayed within the user interaction space of thefirst OS).

Referring back to FIGS. 7 and 8, desktop OS 160 is configured to providea second user interaction space through secondary terminal environment840 suitable to a desktop computing experience. As described above,application 752 may be displayed within the user interaction space ofmobile OS 130 while application 754, running on mobile OS 130, isdisplayed in console window 882 of the second user interaction spacethrough console application 782 running on desktop OS 160 usingembodiments of cross-environment rendering described above. Notably,applications 752 and 754 can be any applications compiled for mobile OS130, running within the mobile OS runtime environment and acceptinginput events through the mobile OS framework without modification forbeing displayed or interacted with remotely (not on the user interactionspace of mobile OS 130).

FIG. 13 illustrates OS architecture configuration 300 b for providinguser interaction support to cross-environment applications, according tovarious embodiments. In OS architecture configuration 300 b, devicedrivers in shared kernel 320 implement the hardware interfaces for I/Odevices 844, 846, and/or 848 that make up secondary terminal environment840. Through the device drivers, input events for these devices appearin devices 1364, 1366, and/or 1368 of I/O devices 1360 of shared kernel320. Because system resources of shared kernel 320 are available to bothmobile OS 130 and desktop OS 160, mobile OS determines whether inputevents on input devices 1364, 1366, and/or 1368 are intended for mobileOS 130. When desktop OS 160 is configured to provide a second userinteraction space (i.e., mobile computing device 110 is docked tosecondary terminal environment 840), mobile OS 130 ignores input eventsfrom input devices associated with the second user interaction space. InOS architecture configuration 300 b, mobile OS 130 ignores input eventsfrom devices 1364, 1366, and/or 1368. In this instance, desktop OS 160accepts input events from devices 1364, 1366, and/or 1368.

Desktop OS 160 processes input events from devices 1364, 1366, and/or1368 and determines how to distribute the events to various windows orGUI elements within desktop OS 160. For example, window system 774 ofdesktop OS 160 may accept input events from devices 1364, 1366, and/or1368. In one embodiment, the graphics system of desktop OS 160 is anX-windows graphics system.

In one instance, the user clicks a button of a pointing device at ascreen location within console window 882 of console application 782. Acorresponding input event appears in device 1368 and desktop OS 160receives and processes the input event. Window system 774 directs theinput event to console application 782 through which application 754,running on mobile OS 130, is displayed using embodiments ofcross-environment rendering. As described above, application 754 is amobile application and instantiates views and other event handlers usinglibraries of mobile OS 130 that receive input from the input queue 736of mobile OS 130 to accept user input. To present the input event toapplication 754 in such a way that application 754 will properlyinterpret the input event, console application 782 maps the input eventto the graphics context of mobile OS 130 and passes the event to theinput queue 736 of mobile OS 130 through virtual input device 1370.Virtual input device 1370 appears to mobile OS 130 as an input devicewith the proper input event protocol for mobile OS 130. In addition,input events are formatted by console application 782 to be relative tothe graphics context of mobile OS 130. Mobile OS 130 associates virtualinput device 1370 with application 754 such that application 754receives the input event from the mobile OS event queue. For example,mobile OS 130 may associate virtual input device 1370 with the virtualdisplay ID of application 754. In this way, application 754 receives andprocesses the input event just as if application 754 was displayed andinteracted with through the user interaction space of mobile OS 130.

FIG. 14 illustrates aspects of user interaction support forcross-environment applications rendered using a non-extended graphicscontext, according to various embodiments. The method 1400 begins atblock 1402, when an input event is received from an input device (e.g.,keyboard 846, pointing device 848) connected to mobile computing device110. As described above, the input event may appear in an input devicein the shared kernel. At block 1404, the mobile OS determines whethermobile computing device 110 is docked and the desktop OS is associatedwith the input device. For example, if mobile computing device 110 isnot docked with a secondary terminal environment and the desktop OS issuspended, the mobile OS may determine that the desktop OS is notassociated with the input device. If the desktop OS is suspended or theinput device is not part of a secondary terminal environment associatedwith the desktop OS, the mobile OS accepts the input command from theinput device at block 1406.

If the desktop OS is not suspended and the input device is part of asecondary terminal environment associated with the desktop OS, themobile OS ignores the input event on the input device and the desktop OSaccepts the input event at block 1408. At block 1410, the desktop OSdistributes the input event to the appropriate window or GUI elementwithin the desktop OS. If the input event is not directed to the consolewindow, the input event is directed to another window or GUI element atblock 1412. If the input event is directed to the console window (e.g.,the user clicks a pointing device within the console window area), theinput event is passed to the console application as an input event atblock 1414.

At block 1416, a virtual input device is generated for input events fromthe console application. The virtual input device may be generated inthe shared kernel or input events may be written directly into themobile OS input devices. At block 1418, the console application maps theinput event to the graphics context of the mobile OS. For example, theconsole application may map the position of an input event within theconsole window of the console application to the position within thegraphics context of the mobile OS. The console application may translatethe input event for an input format or input mode state for the mobileOS. For example, an input format may be different between the desktop OSand the mobile OS. Even with common input formats, the desktop OS andmobile OS may have different input mode states. For example, the desktopOS may be processing input events using a non-touch mode input statewhile the mobile OS is in a touch mode input state. In this instance, aninput event generated by a pointing device in the desktop OS userinteraction space may be translated to appear as a gesture-based eventin the virtual input device.

At block 1420, the virtual input device is associated in the mobile OSwith the virtual display device for the application displayed within theconsole window. For example, multiple applications may be running on themobile OS and displayed within different console windows on the userinteraction space of the desktop OS. Each application displayed within aseparate console window in the user interaction space of the desktop OSis assigned a virtual display ID within the mobile OS. Therefore, whenthe mobile OS receives an input event from a console application of thedesktop OS through a virtual device, the mobile OS can map the virtualdevice to the correct application through the virtual display ID. Atblock 1422, the input event is passed to the application associated withthe virtual display and the application can process the input event asif it occurred through a user interaction space of the mobile OS.Notably, the above method works with any application of mobile OS 130,the application does not need to be specially designed to accept inputevents through the console application of the desktop OS.

Extended Rendering Context Embodiments

Some embodiments handle concurrent user interface support acrossmultiple OSs by establishing extended rendering contexts within thefirst operating system. As discussed above, the first OS (e.g., themobile OS, Android) is typically configured to define a single, activeuser interaction space with a single active rendering context. Noveltechniques are described herein for effectively fooling the first OSinto concurrently handling multiple user interaction spaces by tiling anumber of so-called “context spaces” into a single, extended renderingspace and associating each context space with a different display.Embodiments include techniques for handling the display outputs tomultiple user interaction spaces and techniques for handling the userinteraction in those contexts.

Returning to the example discussed above with reference to thenon-extended rendering context embodiments, suppose again that acalendar application and a word processing application are both compiledfor a first OS (e.g., mobile OS, Android OS) and are both runningconcurrently within the first OS on a mobile device. A second OS (e.g.,a non-mobile OS, like Hydroid) is running concurrently on the mobiledevice using a shared kernel. A user has docked the mobile device with asecond, desktop computing environment, and the desktop computingenvironment is associated with and displaying the user interaction spacefor the second OS. The user desires to interact with the word processingapplication, running on the first OS, through that desktop computingenvironment of the second OS. It is desirable to handle the userinteraction space of the desktop computing environment using the secondOS of the mobile computing environment (i.e., the mobile device) in away that is transparent to the user.

FIG. 15 illustrates aspects of concurrent user interface support acrossmultiple OSs using extended rendering contexts, according to variousembodiments. In application rendering diagram 1500 illustrated in FIG.15, the first application 1510 (e.g., calendar application) calculatesupdates for a first surface 1512 within the first OS. The first surface1512 is stored in a first memory location in a shared memory space bythe first OS. Specifically, the first application 1510 updates the backbuffer 1514 of the first surface 1512. Similarly, the second application1530 (e.g., word processing application) calculates updates for a secondsurface 1532 using the first operating system. Again, the updates by thesecond application 1530 to the second surface 1532 are made to the backbuffer 1534. The second surface 1532 is stored in a second memorylocation in the shared memory space by the first OS.

The first OS determines when surface information is changed andinitiates a rendering sequence. The first OS may perform a single loopover all surfaces, including the first surface 1512 and second surface1532, determining when surface information associated with particularapplications has changed. In the rendering sequence, the first OSdetermines when surface information is changed and swaps the frontbuffer 1516 and back buffer 1514 of the first surface 1512, and thefront buffer 1536 and back buffer 1534 of the second surface 1532. Thefirst OS establishes an extended rendering context 1520 in a thirdmemory location in a shared memory space and renders the first surface1512 into a first context space 1522 of the extended rendering context1520. The first OS renders the second surface 1533 into a second contextspace 1524 of the extended rendering context 1520.

In embodiments, the extended rendering context 1520 may overlap inmemory space with the frame buffer of the first OS. For example, thememory location of the first context space 1522 of the extendedrendering context 1520 may be coextensive with the frame buffer of thefirst OS. The first OS passes a notification to the second OS that thefinal image is rendered at the third memory location. For example, thefirst OS may pass a file descriptor to the shared memory location of thesecond context space 1524 of the extended context 1520 to a consoleapplication of the second OS through an inter-process communicationchannel. The second OS accesses the second portion of the extendedgraphics context at the third memory location to retrieve the renderedgraphics frame 1538 for display in the console window 1540 of the secondOS. For example, the console application of the second OS may map thefile descriptor to its process space and read the rendered graphicsframe from the third memory location for display within the userinteraction space of the second OS. In this way, the second OS displaysthe rendered graphics frame for the second application within its userinteraction space in real-time.

FIG. 16 shows a flow diagram 1600 of an illustrative method forcross-environment rendering using an extended rendering context,according to various embodiments. Embodiments maintain display ofapplication graphics for a first application (e.g., a calendarapplication) and a second application (e.g., a word processingapplication). It is assumed that both applications are compiled for andin active concurrent execution within a first operating system (e.g.,the Android OS), but that a user desires to interact with the secondapplication through a second computing environment associated with asecond OS. Notably, these techniques can be applied in environmentswhere the two OSs are incompatible (e.g., applications compiled for thefirst OS could not be directly executed on the second OS). In someimplementations, as described above, the two OSs are runningindependently and concurrently on a shared kernel of the mobile device.

The method 1600 begins at block 1604 by establishing an extendedrendering context of the first OS. As discussed above, the renderingcontext is typically established according to characteristics of asingle, active display. However, the extended rendering context isestablished to have a first context space associated with the firstapplication and a second context space associated with the secondapplication. The first and second context spaces are non-overlapping.

In some embodiments, the extended rendering context is generated bytiling the active display of the local device (e.g., the display of themobile device having the shared kernel) and any virtual displays (i.e.,displays of the first OS associated with console windows displayedwithin the user interaction space of the second OS) to form what looksto the first OS to be one, large display. Regions of the extendedrendering context are designated as non-overlapping context spaces tomaintain their association with their respective physical or virtualdisplays. Notably, in some implementations, different context spaces mayhave different resolutions or other characteristics. Also, in certainembodiments, context spaces are not contiguous. For example, theextended rendering context is established in such a way that space isleft between each context space that is not assigned to any contextspace.

At block 1608, updates are calculated to surfaces of the firstapplication and the second application using the first operating system.The updated surfaces are then rendered using the first operating system,at block 1612, to generate an extended graphics frame in a shared memorylocation accessible by both the first operating system and a secondoperating system (e.g., which may be running concurrently on a sharedkernel). A first portion of the extended graphics frame is associatedwith the first context space (associated with the first application) anda second portion of the extended graphics frame is associated with thesecond context space (associated with the second application. When therendering occurs at block 1608, the updated surfaces of the firstapplication are rendered to the first portion of the extended graphicsframe, and the updated surfaces of the second application are renderedto the second portion of the extended graphics frame. It is worth notingthat, in this way, the extended graphics frame effectively includesrendered surfaces of both applications tiled into their appropriatecontext spaces.

In some embodiments, at block 1616, the first portion of the extendedgraphics frame associated with the first context space is displayed fromthe shared memory location to a display(s) of a first computingenvironment associated with the first OS. For example, as discussedabove, the shared memory location is frame buffer memory (or is copiedto frame buffer memory) of the mobile device, and a display devicedriver of the mobile device accesses the frame buffer memory to displaythe first portion of the extended graphics frame to its display(s).Further, in some embodiments, the second portion of the extendedgraphics frame associated with the second motion space is displayed, atblock 1620, from the shared memory location to a display of a secondcomputing environment associated with the second operating system. Forexample, as discussed above, the shared memory location is copied toframe buffer memory of the second OS associated with the second (e.g.,desktop) computing environment, and a display device driver displays thesecond portion of the extended graphics frame to a display of the secondcomputing environment.

In embodiments discussed with reference to FIG. 12, rendering for thesecond application's updated graphics is performed remotely by thesecond OS. In embodiments discussed with reference to FIG. 15, renderingfor both applications is performed locally by the rendering engine ofthe mobile device, but the rendering context is continually establishedand disestablished. The embodiments discussed with reference to FIG. 16allow rendering for both applications to be performed locally by therendering engine of the mobile device, while maintaining a single,albeit extended, rendering context (i.e., without disestablishing therendering context).

FIG. 17 shows a flow diagram 1700 of another illustrative method forcross-environment rendering using an extended rendering context,according to various embodiments. As in the method 1600 of FIG. 16,embodiments maintain display of application graphics for a firstapplication and a second application that are both compiled for and inactive concurrent execution within a first operating system. The method1700 begins by establishing an extended rendering context of the firstoperating system at block 1704 and calculating updates to surfaces ofthe first application and the second application using the firstoperating system at block 1708. As discussed above, the extendedrendering context is established to have a first context spaceassociated with the first application and a second context spaceassociated with the second application. The first and second contextspaces are non-overlapping. In some implementations, blocks 1704 and1708 are performed in a substantially identical manner to blocks 1604and 1608, respectively.

At block 1712, the updated surfaces of the first application arerendered according to the first context space using the first operatingsystem to generate a first graphics frame in a frame buffer of the firstoperating system. For example, the first context space may be associatedwith a particular resolution, particular tiling offsets (e.g., starting“X” position), etc. In some implementations, the first graphics frame isgenerated in a substantially identical manner to generation of therespective portion of the extended graphics frame in the method 1600 ofFIG. 16. In some embodiments, at block 1716, the first graphics frameassociated with the first context space is displayed from the framebuffer to a display of a first computing environment associated with thefirst operating system.

At block 1720, the updated surfaces of the second application arerendered using the first operating system to generate a second graphicsframe in a shared memory location. As discussed above, the shared memorylocation is accessible by both the first operating system and a secondoperating system (e.g., which may be running concurrently on a sharedkernel). In some embodiments, at block 1724, the second graphics frameassociated with the second motion space is displayed from the sharedmemory location to a display of a second computing environmentassociated with the second operating system.

Notably, the embodiments of both FIGS. 16 and 17 establish extendedrendering contexts with context spaces for each application. However,while the method 1600 of FIG. 16 renders all the graphics updates into asingle extended bitmap, the method 1700 of FIG. 17 renders the graphicsupdates into separate bitmaps. One or the other technique may bedesirable, for example, depending on how memory is being managed and/oraccessed.

As with the embodiment of FIG. 12, in the embodiments of FIGS. 16 and 17both applications' updated graphics frames are rendered using the firstOS's rendering engine. Using the first OS's rendering engine allows bothapplications to use hardware acceleration capabilities of the mobiledevice that are available in the first OS. For example, in theembodiments of FIGS. 12, 16, and/or 17, either or both of the first andthe second application may be rendered by the first OS using 2D or 3Dhardware-assisted rendering.

Remote display of a graphics frame for an application running on a firstOS (i.e., mobile OS, Android) using an extended rendering contextprovides a way for the first OS to provide rendering for multipleapplications for display within multiple user interaction spaces using asingle rendering context. However, an extended rendering context createsissues for handling input events for applications displayed through theextended rendering context. Specifically, the input queue of the firstOS must be configured to handle multiple input events from multipleapplications displayed through separate virtual displays of an extendedrendering context.

Embodiments of cross-environment user interaction support are directedto handling user input events for multiple applications running on afirst OS and displayed on multiple separate user interaction spaces(i.e., the mobile device user interaction space and a desktop OS userinteraction space) through an extended rendering context of the firstOS. Embodiments include an extended input queue where input events fromvirtual input devices for remotely displayed applications are mapped toseparate motion spaces within the input queue. For example, a firstapplication (e.g., calendar application) is running on a first OS and isbeing displayed to a first display associated with the first device(e.g., the display of the mobile device on which the first OS isrunning) A second application (e.g., word processing application) isalso running concurrently with the first application, but is renderedwithin a context space (i.e., virtual display) of the extended renderingcontext and displayed on a second user interaction space of a second OSrunning concurrently with the first OS on a shared kernel of a mobiledevice. The first OS renders a graphics frame through an extendedrendering context that includes both application graphics for the firstapplication in the first context space (i.e., the mobile device display)and the second application in the second context space. The secondcontext space is displayed on a user interaction space of the second OSthrough a console application running on the second OS.

Input events for applications displayed remotely through an extendedrendering context are received by the second OS (i.e., desktop OS,Hydroid) and passed to a virtual input device by the console applicationof the second OS in the same manner as described above for non-extendedgraphics contexts. However, as described above, the input eventsreceived in the mobile OS from the virtual input device are relative tothe console window displayed within the user interaction space of thesecond OS. Virtual input devices are mapped to motion spaces within theextended input queue that are associated with virtual displayscorresponding to remotely displayed applications. The extended inputqueue allows the first OS to correctly process input from multiple localand virtual input devices intended for multiple concurrently executingapplications using a single input queue.

FIGS. 18 a and 18 b illustrate aspects of user interaction support forcross-environment applications using an extended rendering context,according to various embodiments. FIG. 18 a illustrates a userinteraction space 1810 that is remotely displaying applications runningon a first OS (e.g., the GUI of the second OS displaying applicationsrunning in the first OS). For example, first, second, and thirdapplications may be running on the first OS (i.e., in active concurrentexecution with the first OS). The first OS may display the first,second, and third applications within a first, second, and third contextspace of an extended rendering context of the first OS according toembodiments described above. Console windows 1812 and 1814 in userinteraction space 1810 may be displaying the second application and thethird application running in the first OS, respectively.

FIG. 18 b illustrates an extended input queue 1840 of the first OS thatprovides user interaction support for each application running on thefirst OS. The extended input queue 1840 includes a first motion space1842, a second motion space 1844, and a third motion space 1846. Thefirst motion space is associated with the first context space of theextended rendering context of the first OS, which is typically used torender a non-virtual display 1852 of the first OS (i.e., the contextspace associated with display(s) 116 of mobile computing device 110).The second and third motion spaces are associated with virtual displays1854, 1856, which are rendered through the second and third contextspaces, respectively.

When an input event occurs that is directed to a console window of aremotely displayed application, the input event is directed to themotion space associated with the virtual display through which theapplication is displayed. For example, if the user clicks with apointing device within console window 1812 of user interaction space1810, as indicated by input event 1820, the window system of the secondOS directs the input event to the console application associated withconsole window 1812. The console application maps the input event to avirtual input device as described above. However, the input event isrelative to the console window 1812. If the input event is fed directlyto the input queue of the first OS, the input event would not bedirected to the correct application event handler. Therefore, the inputevent 1820 from the virtual input device is remapped to the secondmotion space 1844. In this way, the extended input queue directs theinput event to event handlers of the second application which receiveand process the input event 1820.

In embodiments, virtual displays are offset within the mobile OS inputqueue. For example, in FIG. 18 b, virtual displays 1854 and 1856 areoffset by virtual display offset 1858 within the mobile OS input queue1840. Virtual display offset 1858 prevents virtual displays fromappearing adjacent within the input queue, which may cause an inputevent intended for one virtual display from being interpreted within amotion space associated with a different application. The virtualdisplay offset should be large enough never to be used as an actualvirtual display resolution parameter. In one embodiment, virtual displayoffset 1858 is selected to be 10000 pixels.

FIG. 19 illustrates a method 1900 for receiving input events forcross-environment applications displayed through an extended renderingcontext, according to various embodiments. For example, method 1900 maybe used to process input events for a first application and a secondapplication running within a first OS, the first application displayedlocally on the user interaction space of the first OS and the secondapplication displayed remotely in a user interaction space of a secondOS through an extended rendering context of the first OS.

Method 1900 begins at block 1902, when a first user input is received ina first OS, a first application and a second application in activeconcurrent execution within the first OS, the first applicationdisplayed within a first user environment associated with the first OSand the second application displayed within a second user environmentassociated with a second OS, the first and second operating systemsrunning concurrently on a shared kernel, the first OS maintainingapplication graphics for the second application by rendering a graphicsframe for the second application through a first virtual display of anextended rendering context. At block 1904, the first OS establishes anextended input queue that includes a first motion space and a secondmotion space, the second motion space corresponding to the first virtualdisplay. For example, the first operating system allocates the firstvirtual display for the second application, establishes an extendedrendering context having a first context space and a second contextspace, associates the first virtual display with the second contextspace, and renders a graphics frame for the second application throughthe second context space of the extended rendering context using thetechniques described above.

At block 1906, a first user input event is received by the first OS at afirst virtual input device from a first console application running inthe second OS that is displaying the rendered graphics frame for thesecond application through the second OS. At block 1908, the firstvirtual input device is mapped to the second motion space of theextended input queue of the first operating system. Mapping the firstvirtual input device to the second motion space allows the extendedinput queue of the first OS to correctly associate input events from thefirst virtual input device to event handlers within views of the secondapplication. Specifically, when the input event is mapped to the secondmotion space, the first OS will treat the input event as occurring at alocation associated with the second application in the extended inputqueue. At block 1910, the first OS passes the first user input event tothe second application from the mapped first virtual input device. Theextended input queue uses the tiled nature of the extended renderingcontext to enable the input queue to handle multiple input events frommultiple user interaction spaces and direct the input events to theappropriate event handlers of the intended applications.

Mirrored Context Embodiments

Embodiments of the extended and non-extended rendering contexts aredescribed above in the context of maintaining concurrent userinteraction space support across multiple applications over multipleoperating systems. In many instances, it is desirable to mirror thecontext for a single user interaction space. It is desired to view andinteract with the first OS (i.e., to “mirror” the interaction space)concurrently in a second computing environment associated with a secondOS (e.g., a desktop environment associated with Hydriod OS). Through themirrored user interaction space, the user can interact with the first OSas if interacting through the local device (i.e., the user can browseavailable applications, start and stop applications, use the searchcapabilities of the first OS, etc.).

The first OS (e.g., the mobile OS, Android) is typically configured todefine a single, active user interaction space. The user interactionspace includes an active display (e.g., with associated characteristics,like resolution) and one or more active input devices for allowing userinteraction with the elements displayed on the active display. Noveltechniques are presented for using cross-environment rendering toprovide one or more mirrored user interaction spaces across multipleOSs. As discussed above, embodiments operate even where the multiple OSsare incompatible and/or are running independently and concurrently on ashared kernel.

Maintaining concurrent user interaction support with a mirrored contextmay be accomplished using many of the same system elements referred toabove with regard to maintaining concurrent user interaction support forcross-environment applications. For example, referring to FIG. 7, thegraphics context for mobile OS 130 may be actively displaying anapplication (e.g., applications 752 and/or 754) and/or a home screen ofthe mobile OS 130 (e.g., home screen application 451 of Android OS 430).Surface information for an actively displayed application and/or thehome screen of the mobile OS may be stored within shared memory 724. Themirrored context for mobile OS 130 may be displayed within the userinteraction space of desktop OS 160 through console application 782.

FIG. 20 shows a flow diagram 2000 of an illustrative method forcross-environment rendering of a graphics context to provide a mirroreduser interaction space, according to various embodiments. The method2000 begins at block 2004 by calculating, using a first operatingsystem, updates to a set of surfaces of a first application compiled forand in active execution within the first operating system. For example,calculations are made to determine changes in surface shapes, sizes,textures, layering, etc. The surface updates are then rendered at block2008, using the first operating system, to generate a graphics frame.The graphics frame may be a bitmap that reflects the updated graphicsinformation for the application.

At block 2012, the graphics frame is stored in a shared memory locationaccessible by both the first operating system and a second operatingsystem. In some embodiments, the first and second OS are runningconcurrently on a shared kernel. The graphics frame may be displayed toa first application display of the first application on a first displayof a first computing environment using the first operating system atblock 2016. For example, the shared memory location may be frame buffermemory or may be copied to a frame buffer of the first operating system.A display device driver of the local device (e.g., which is running theshared kernel) accesses the frame buffer memory to display the bitmap.

Subsequent to storing the graphics frame in the shared memory locationat block 2012, it is desirable to inform the second OS that the updatedgraphics information is available. At block 2020, a file descriptor ispassed, indicating the shared memory location to a console applicationcompiled for and in active execution within the second OS. In someimplementations, the file descriptor includes an indication of theshared memory location. In other implementations, the file descriptorincludes additional information, like a flag indicating availability ofupdated graphics information for the application being mirrored.

As described above, the console application may be an X-Windows orsimilar type of application that is displayed within a window of adisplay in the second computing environment. At block 2024, the consoleapplication accesses the updated graphics information (e.g., the bitmap)at the shared memory location according to the file descriptor anddisplays the graphics frame from the shared memory location to a secondapplication display of the first application on a second display of asecond computing environment. In some embodiments, the updated graphicsinformation of the application is displayed substantially concurrentlyon the displays of both the first and second computing environments.

FIG. 21 shows a flow diagram 2100 of another illustrative method forcross-environment rendering of a graphics context to provide a mirroreduser interaction space, according to various embodiments. As in FIG. 20,the method 2100 begins at block 2104 by calculating, using a firstoperating system, updates to a set of surfaces of a first applicationcompiled for and in active execution within the first operating system.At block 2108, the updated set of surfaces is stored in a shared memorylocation accessible by both the first operating system and a secondoperating system (e.g., running concurrently on a shared kernel).

At block 2112, the updated set of surfaces is rendered with the firstoperating system to generate a first graphics frame. The first graphicsframe can then be displayed, at block 2116, to a first applicationdisplay of the first application on a first display of a first computingenvironment using the first operating system. For example, the mobile OS130 renders the updated application graphics and displays the updatedgraphics to the display(s) 116 of the mobile device 110.

At any time subsequent to storing the updated set of surfaces in sharedmemory in block 2108, it is desirable to notify the second OS that theupdated graphics information is available. At block 2120, a filedescriptor is passed indicating the shared memory location to a consoleapplication compiled for and in active execution within the secondoperating system. Notably, the information stored in the shared memoryis un-rendered surface information (e.g., geometric primitives) ratherthan rendered bits as in the method 2000 of FIG. 20.

Accordingly, at block 2124, the updated set of surfaces is rendered bythe second operating system (e.g., via the console application accordingto the file descriptor) from the shared memory location to generate asecond graphics frame that is substantially identical to the firstgraphics frame. At block 2128, the second graphics frame is displayed toa second application display of the first application on a seconddisplay of a second computing environment via the console application ofthe second operating system, such that the second application display issubstantially identical to the first application display.

It is worth noting that additional overhead may be involved inreplicating the rendering on both the first and second operatingsystems. However, this additional overhead may be worthwhile in a numberof circumstances. For example, where the displays of the differentcomputing environments have appreciably different characteristics, itmay be desirable to render updated graphics information in separaterendering contexts that are each suited for a respective one of thedisplays.

The methods of FIGS. 20 and 21 describe cross-environment mirroring of agraphics context with active display of an application running in thefirst OS within the mirrored graphics context. However, the methods maybe used where no application is actively displayed within the graphicscontext. For example, the graphics context may be displaying a homescreen or other feature (e.g., search screen, etc.) of the first OS. Inthese instances, the surface information for the graphics context isupdated by a component of the first OS, and the other steps of themethods of FIGS. 20 and 21 may be performed to provide the mirroredgraphics context in the second application display.

Notably, cross-environment mirroring of a graphics context may beemployed concurrently with cross-environment rendering of anapplication. For example, a method according to FIG. 20 or 21 may beused to mirror the active graphics context of the user interaction spaceof the mobile device to a second user environment at the same time thatan application running on the first OS is displayed within the seconduser environment using the techniques for cross-environment rendering ofan application described above. Referring to FIG. 8 for the sake ofillustration, the user interaction space of the mobile OS may bedisplayed within a first console window 882 while a mobile OSapplication is displayed within a second console window 884 within theuser interaction space of the desktop OS on display 844.

Providing user interaction support for a mirrored graphics context maybe performed in substantially the same way as providing user interfacesupport for a cross-environment application illustrated in FIGS. 13, 14,18, and/or 19. Specifically, input events may be provided from a consoleapplication of the second OS to a virtual input device. The first OS mayaccept input events from the virtual input device through an extended ora non-extended input queue.

Cross-Environment Redirection Embodiments

The techniques described above provide cross-environment userinteraction support for applications and graphics contexts of a firstoperating system through a user interaction space of a second operatingsystem. To facilitate a transparent cross-environment use model,embodiments are directed to providing access to applications and/ormirrored contexts of a first operating system from the user interactionspace of the second operating system.

Referring back to FIG. 8, a user may interact with a first OS (i.e.,mobile OS, Android) through a first user interaction space that includesthe interaction components (i.e., touch screen display(s) 116, other I/Odevice(s) 118) on the mobile device. The user may also interact with asecond OS (i.e., desktop OS, Hydroid) through a second user interactionspace including a display 844 of secondary terminal environment 840. Asdescribed above, the set of applications available for (i.e., compiledfor and loaded within the execution environment of) desktop OS 160 maybe different than that available for mobile OS 130. Embodiments aredirected to making applications of mobile OS 130 accessible within theuser interaction space of desktop OS 160 by providing menu icons or menulist items within menus of the user interaction space of desktop OS 160for applications available on mobile OS 130.

FIG. 22 illustrates aspects of cross-environment redirection, accordingto various embodiments. Within computing environment 2200 illustrated inFIG. 22, a user interacts with desktop OS 160 through desktop OS userinteraction space 2202. Within desktop OS user interaction space 2202,menu bar 2220 includes icons or lists of available applications. Tolaunch an application, the user selects the application name or iconfrom the menu bar or from drop-down or pop-up-lists of menu bar 2220.Traditionally, menu bar 2220 includes only menu items or icons forapplications available on desktop OS 160. For example, menu items 2222,2224, 2226, and/or 2228 may be applications available on (i.e., compiledfor and loaded within the execution environment of) desktop OS 160.Embodiments of the invention are directed to providing cross-environmentaccess to applications and/or the graphics context of mobile OS 130 fromdesktop OS user interaction space 2202. For example, menu items 2232,2234, 2236, 2237, and/or 2238 may indicate applications available onmobile OS 130 and/or the graphics context of mobile OS 130.

Desktop OS user interaction space 2202 is displayed on a display withina user interaction space (e.g., secondary terminal environment 840)associated with desktop OS 160. Menu bar 2220 of desktop OS 160 includesmenu items 2222, 2224, 2226, and/or 2228 associated with applicationscompiled for and loaded on desktop OS 160 (e.g., compiled forHydroid/Linux and loaded within the execution environment of HydroidOS). Menu bar 2220 also includes menu items 2234, 2236, 2237, and/or2238 associated with applications compiled for and loaded on mobile OS130 (e.g., compiled for Android and loaded within the Android executionenvironment). When the user selects one of menu items 2234, 2236, and/or2238, the associated application is launched on mobile OS 130 anddisplayed within a console window of desktop OS 160, for example, withinwindow 2216 of desktop OS user interaction space 2202. Menu item 2232may be associated with the mobile OS graphics context such that if menuitem 2232 is selected, the graphics context of the mobile OS isdisplayed within a console window of desktop OS 160.

FIG. 23 illustrates process flow 2300 that may be employed to build menubar 2220 of desktop OS 160. At step 2302 of process flow 2300, desktopOS 160 queries mobile OS 130 for a list of available applications. Inone embodiment, a system service or launcher application of desktop OS160 queries a service of mobile OS 130 for all launchable mobile OSapplication shortcuts. Mobile OS 130 responds with a list ofapplications that are available (i.e., launchable shortcuts foravailable mobile OS applications). The list of available applicationsmay include all applications available on mobile OS 130 (allapplications loaded and executable on mobile OS 130) or a subset ofavailable mobile OS applications. For example, the list may include allapplications that appear on the application menu screen(s) of the mobileOS GUI. At step 2304, desktop OS 160 receives the list of applicationsfrom mobile OS 130. The list of applications returned by mobile OS 130includes application package names for each listed application, and mayalso include application names and icons for each listed application.

Desktop OS 160 creates the menu items in menu bar 2220 for eachapplication of the list of applications by iterating over blocks 2306,2308, and 2310. For each application, desktop OS 160 instantiates anicon for the application in menu bar 2220 at block 2306, associates theicon with a console application of desktop OS 160 at block 2308, andassociates a parameter that indicates the package name of theapplication with the icon at block 2310. The console application runs ondesktop OS 160 and displays graphics information for the applicationwithin desktop OS 160, using embodiments of cross-environment renderingdescribed above. In this way, when a user selects the menu item, theconsole application is launched on desktop OS 160, and the package nameof the application is passed to the console application.

Desktop OS 160 may display the menu items associated with applicationsof mobile OS 130 in a variety of ways. The menu items may be displayedin menu bar 2220, or a drop-down menu that appears when a menu item thatindicates that mobile OS applications are available is selected. Themenu items may be displayed using icons or only application names inmenu bar 2220 or the drop-down menu. In one embodiment, desktop OS 160displays a separate menu bar for mobile OS applications. In anotherembodiment, menu items associated with mobile OS applications appearwithin the desktop OS menu bar 2220 alongside or intermingled with menuitems for desktop OS applications. Optionally, the mobile OS menu itemsmay be in an area 2230 of menu bar 2220 set off by delimiter 2240 orotherwise identifiable as including mobile OS menu items. Menu bar 2220may include a menu item for the active display of the mobile deviceitself, i.e., a menu item that the user can select to display the userinteraction space of the mobile OS within a user environment of thedesktop OS according to the methods of FIGS. 20 and/or 21. In oneembodiment, the home screen application of the mobile OS is returned inthe list of available applications and provided an associated menu item.

When the user selects a menu item associated with a mobile OSapplication, desktop OS 160 launches the console application associatedwith the menu item and passes the package name of the application to theconsole application. The console application displays a window withinthe desktop OS user interaction space 2202 (i.e., the consoleapplication displays within the graphics system of the desktop OS). Theconsole application sends a request to mobile OS 130 to launch theapplication (i.e., requests mobile OS 130 to launch the applicationpackage name provided to the console application as an executionparameter) and display the graphics frame for the application throughthe console application. The application may or may not be currentlyrunning on mobile OS 130. If the application is currently running onmobile OS 130, the display of the application may be moved from mobileOS 130 to the desktop OS user interaction space 2202 or displayed bothon a display of the mobile device and user interaction space 2202 at thesame time. Display of application graphics and user interaction supportmay be accomplished for the application using any of thecross-environment rendering and cross-environment user interface supporttechniques described above.

FIG. 24 illustrates process flow 2400 followed by mobile OS 130 tolaunch an application in response to the user selecting a menu itemassociated with a mobile OS application on menu bar 2220 of desktop OSGUI 880. Process flow 2400 begins at block 2402 when mobile OS 130receives the request from desktop OS 160 to launch an applicationcompiled for the mobile OS and loaded within the execution environmentof the mobile OS for display on the desktop OS. At block 2404, themobile OS allocates an unused virtual display ID. For example, thegraphics system of the mobile OS may keep a list of virtual display ID'sand allocate an unused virtual display ID to the process of the firstapplication. At block 2406, the mobile OS launches the first applicationwithin the mobile OS (i.e., running on the mobile OS). At block 2408,mobile OS 130 associates refresh notifications for the first applicationwith the virtual display. For example, the graphics server of the mobileOS may keep a list of applications and their associated virtualdisplays. At blocks 2410 and 2412, the mobile OS maintains graphicsinformation for the first application by monitoring application graphicsfor the first application and notifying a console application of thedesktop OS when application graphics information for the firstapplication is updated. Blocks 2410 and 2412 may correspond tomaintaining application graphics for cross-environment applicationsaccording to the methods of FIGS. 10, 12, 16 and/or 17.

The foregoing description has been presented for purposes ofillustration and description. Furthermore, the description is notintended to limit embodiments of the invention to the form disclosedherein. While a number of exemplary aspects and embodiments have beendiscussed above, those of skill in the art will recognize certainvariations, modifications, permutations, additions, and sub-combinationsthereof.

The various operations of methods described above may be performed byany suitable means capable of performing the corresponding functions.The means may include various hardware and/or software component(s)and/or module(s), including, but not limited to a circuit, anapplication specific integrated circuit (ASIC), or processor.

The various illustrative logical blocks, modules, and circuits describedmay be implemented or performed with a general purpose processor, adigital signal processor (DSP), an ASIC, a field programmable gate arraysignal (FPGA), or other programmable logic device (PLD), discrete gate,or transistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. A generalpurpose processor may be a microprocessor, but in the alternative, theprocessor may be any commercially available processor, controller,microcontroller, or state machine. A processor may also be implementedas a combination of computing devices, e.g., a combination of a DSP anda microprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration.

The steps of a method or algorithm described in connection with thepresent disclosure, may be embodied directly in hardware, in a softwaremodule executed by a processor, or in a combination of the two. Asoftware module may reside in any form of tangible storage medium. Someexamples of storage media that may be used include random access memory(RAM), read only memory (ROM), flash memory, EPROM memory, EEPROMmemory, registers, a hard disk, a removable disk, a CD-ROM and so forth.A storage medium may be coupled to a processor such that the processorcan read information from, and write information to, the storage medium.In the alternative, the storage medium may be integral to the processor.A software module may be a single instruction, or many instructions, andmay be distributed over several different code segments, among differentprograms, and across multiple storage media.

The methods disclosed herein comprise one or more actions for achievingthe described method. The method and/or actions may be interchanged withone another without departing from the scope of the claims. In otherwords, unless a specific order of actions is specified, the order and/oruse of specific actions may be modified without departing from the scopeof the claims.

The functions described may be implemented in hardware, software,firmware, or any combination thereof. If implemented in software, thefunctions may be stored as one or more instructions on a tangiblecomputer-readable medium. A storage medium may be any available tangiblemedium that can be accessed by a computer. By way of example, and notlimitation, such computer-readable media can comprise RAM, ROM, EEPROM,CD-ROM, or other optical disk storage, magnetic disk storage, or othermagnetic storage devices, or any other tangible medium that can be usedto carry or store desired program code in the form of instructions ordata structures and that can be accessed by a computer. Disk and disc,as used herein, include compact disc (CD), laser disc, optical disc,digital versatile disc (DVD), floppy disk, and Blu-ray® disc where disksusually reproduce data magnetically, while discs reproduce dataoptically with lasers.

Thus, a computer program product may perform operations presentedherein. For example, such a computer program product may be a computerreadable tangible medium having instructions tangibly stored (and/orencoded) thereon, the instructions being executable by one or moreprocessors to perform the operations described herein. The computerprogram product may include packaging material.

Software or instructions may also be transmitted over a transmissionmedium. For example, software may be transmitted from a website, server,or other remote source using a transmission medium such as a coaxialcable, fiber optic cable, twisted pair, digital subscriber line (DSL),or wireless technology such as infrared, radio, or microwave.

Further, modules and/or other appropriate means for performing themethods and techniques described herein can be downloaded and/orotherwise obtained by a user terminal and/or base station as applicable.For example, such a device can be coupled to a server to facilitate thetransfer of means for performing the methods described herein.Alternatively, various methods described herein can be provided viastorage means (e.g., RAM, ROM, a physical storage medium such as a CD orfloppy disk, etc.), such that a user terminal and/or base station canobtain the various methods upon coupling or providing the storage meansto the device. Moreover, any other suitable technique for providing themethods and techniques described herein to a device can be utilized.

Other examples and implementations are within the scope and spirit ofthe disclosure and appended claims. For example, due to the nature ofsoftware, functions described above can be implemented using softwareexecuted by a processor, hardware, firmware, hardwiring, or combinationsof any of these. Features implementing functions may also be physicallylocated at various positions, including being distributed such thatportions of functions are implemented at different physical locations.Also, as used herein, including in the claims, “or” as used in a list ofitems prefaced by “at least one of” indicates a disjunctive list suchthat, for example, a list of “at least one of A, B, or C” means A or Bor C or AB or AC or BC or ABC (i.e., A and B and C). Further, the term“exemplary” does not mean that the described example is preferred orbetter than other examples.

Various changes, substitutions, and alterations to the techniquesdescribed herein can be made without departing from the technology ofthe teachings as defined by the appended claims. Moreover, the scope ofthe disclosure and claims is not limited to the particular aspects ofthe process, machine, manufacture, composition of matter, means,methods, and actions described above. Processes, machines, manufacture,compositions of matter, means, methods, or actions, presently existingor later to be developed, that perform substantially the same functionor achieve substantially the same result as the corresponding aspectsdescribed herein may be utilized. Accordingly, the appended claimsinclude within their scope such processes, machines, manufacture,compositions of matter, means, methods, or actions.

What is claimed is:
 1. A method, comprising: receiving a first userinput event in a first operating system, a first application and asecond application in active concurrent execution within the firstoperating system, the first application displayed within a first userenvironment associated with the first operating system and the secondapplication displayed within a second user environment associated with asecond operating system, the first operating system maintainingapplication graphics for the second application by rendering a graphicsframe for the second application through a first virtual display of anextended rendering context, the graphics frame in a first memorylocation of anonymous shared memory comprising named memory blocks, theanonymous shared memory accessible by name through a file descriptor byall processes running on the first operating system and the secondoperating system and not accessible by name by other processes, thememory blocks freed by a single shared kernel, the first operatingsystem and the second operating system running concurrently on thesingle shared kernel; establishing an extended input queue of the firstoperating system having a first motion space and a second motion space,the second motion space associated with the first virtual display;receiving the first user input event at a first virtual input devicefrom the first console application of the second operating system;mapping the first virtual input device to the second motion space of theextended input queue of the first operating system; and passing thefirst user input event to the second application from the mapped firstvirtual input device, wherein the first user environment is a mobiledevice with a first computing environment and the second userenvironment is a desktop computing system with a second computingenvironment, wherein the first operating system and the second operatingsystem execute on the mobile device, wherein the single shared kernelincludes an inter-process communications driver which passes the filedescriptor to processes in the first operating system and the secondoperating system to allow communication between the first operatingsystem and the second operating system so as to communicate acrossprocess boundaries, wherein the mobile device and the desktop computingsystem are distinct computing devices.
 2. The method of claim 1, whereina third application is in active concurrent execution with the first andsecond applications within the first operating system, the firstoperating system rendering application graphics for the thirdapplication to a second virtual display, wherein the first virtualdisplay and the second virtual display are not adjacent to each other inthe extended input queue.
 3. The method of claim 2, wherein the firstvirtual display and the second virtual display are offset within theextended input queue by a virtual display offset that is larger than aresolution of virtual displays of the first operating system.
 4. Themethod of claim 2, wherein a second console application of the secondoperating system displays application graphics for the third applicationrendered by the first operating system to the second virtual display. 5.The method of claim 4, further comprising: receiving a second user inputcommand at a second virtual input device from the second consoleapplication; mapping the second virtual device to a third motion spaceof the extended input queue of the first operating system; and passingthe second user input command to the third application from the mappedsecond virtual input device.
 6. The method of claim 1, wherein the firstconsole application translates the first user input event from a firstinput event protocol of the window system of the second operating systemto a second input event protocol of the first operating system.
 7. Themethod of claim 1, wherein the first operating system comprises a mobileoperating system and the second operating system comprises a desktopoperating system.
 8. A mobile computing device, comprising: a firstapplication and a second application in active concurrent executionwithin a first operating system; a first input device that receives afirst input event from an input element of a first user environment, thefirst user environment associated with the first operating system; asecond input device that receives a second input event from an inputelement of a second user environment, the second user environmentassociated with a second operating system; anonymous shared memorycomprising named memory blocks accessible by name through a filedescriptor by all processes running on the first operating system andthe second operating system and not accessible by name by otherprocesses, the memory blocks freed by a single shared kernel, the firstoperating system and the second operating system running concurrently onthe single shared kernel; a console application running in the secondoperating system that receives the second input event and passes a thirdinput event based on the second input event to a virtual input deviceaccessible by the first operating system; and an extended input queue ofthe first operating system, the extended input queue including a firstmotion space associated with a local display of the first operatingsystem and a second motion space associated with a first virtual displayof the first operating system, the first virtual display associated withthe second application, wherein the first operating system maps thevirtual input device to the second motion space, wherein the first userenvironment is the mobile device and the second user environment is adesktop computing system, wherein the first operating system and thesecond operating system execute on the mobile device, wherein the singleshared kernel includes an inter-process communications driver whichpasses the file descriptor to processes in the first operating systemand the second operating system to allow communication between the firstoperating system and the second operating system so as to communicateacross process boundaries, wherein the mobile device and the desktopcomputing system are distinct computing devices.
 9. The mobile computingdevice of claim 8, wherein a graphics server of the second operatingsystem is a desktop type graphics server and wherein the secondapplication uses a graphics library of the first operating system thatis incompatible with the desktop type graphics server of the secondoperating system.
 10. A computing device including a non-transitorycomputer-readable medium storing instructions for a physical processor,the instructions, when executed, causing the processor to perform stepscomprising: receiving a first user input event in a first operatingsystem, a first application and a second application in activeconcurrent execution within the first operating system, the firstapplication displayed within a first user environment associated withthe first operating system and the second application displayed within asecond user environment associated with a second operating system, thefirst operating system maintaining application graphics for the secondapplication by rendering a graphics frame for the second applicationthrough a first virtual display of an extended rendering context, thegraphics frame in a first memory location of anonymous shared memory,the anonymous shared memory comprising named memory blocks accessible byname through a file descriptor by all processes running on the firstoperating system and the second operating system and not accessible byname by other processes, the memory blocks freed by a single sharedkernel, the first operating system and the second operating systemrunning concurrently on the single shared kernel; establishing anextended input queue of the first operating system having a first motionspace and a second motion space, the second motion space associated withthe first virtual display; receiving the first user input event at afirst virtual input device from the first console application of thesecond operating system; mapping the first virtual input device to thesecond motion space of the extended input queue of the first operatingsystem; and passing the first user input event to the second applicationfrom the mapped first virtual input device, wherein the first userenvironment is the mobile device and the second user environment is adesktop computing system, wherein the first operating system and thesecond operating system execute on the mobile device.
 11. The computingdevice of claim 10, wherein a third application is in active concurrentexecution with the first and second applications within the firstoperating system, the first operating system rendering applicationgraphics for the third application to a second virtual display, whereinthe first virtual display and the second virtual display are notadjacent to each other in the extended input queue.
 12. The computingdevice of claim 11, wherein the first virtual display and the secondvirtual display are offset within the extended input queue by a virtualdisplay offset that is larger than a resolution of virtual displays ofthe first operating system.
 13. The computing device of claim 11,wherein a second console application of the second operating systemdisplays application graphics for the third application rendered by thefirst operating system to the second virtual display.
 14. The computingdevice of claim 13, further comprising: receiving a second user inputcommand at a second virtual input device from the second consoleapplication; mapping the second virtual device to a third motion spaceof the extended input queue of the first operating system; and passingthe second user input command to the third application from the mappedsecond virtual input device.
 15. The computing device of claim 10,wherein the first console application translates the first user inputevent from a first input event protocol of the window system of thesecond operating system to a second input event protocol of the firstoperating system.
 16. The computing device of claim 10, wherein thefirst operating system comprises a mobile operating system and thesecond operating system comprises a desktop operating system.
 17. Thecomputing device of claim 10, wherein single shared kernel includes aninter-process communications driver which passes the file descriptor toprocesses in the first operating system and the second operating systemto allow communication between the first operating system and the secondoperating system so as to communicate across process boundaries.
 18. Thecomputing device of claim 17, wherein the mobile device and the desktopcomputing system are distinct computing devices.